Pith. sign in

REVIEW 3 major objections 6 minor 26 references

VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces VibeLifeBench, a 200-task benchmark that places agents in multi-week simulated life worlds with silent changes, and reports that seven frontier models all score low, with the best at an average of 32.5 out of 100.

desk verdict Solid new benchmark for proactive living-world agents, but the claim that frontier models are 'far from able' needs an oracle or human baseline to be trusted. read the letter →

arxiv 2608.10875 v1 pith:PFOPO3BY submitted 2026-08-11 cs.CL cs.AI

Xiaohongshu Inc This is my paper · ORCID
classification cs.CLcs.AI
keywords LLMagentslifeassistanceproactivelong-horizontasksliving-worldsimulationagentevaluationsilentmutationsbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VibeLifeBench is built around a simple reframing: a life-assistance task is not a prompt but a world with a clock. Each of its 200 tasks runs a multi-week simulated life scenario across 22 mock services, and about a fifth of the 7,453 scripted events are silent background mutations that fire with no notification, so an agent that never re-checks the world cannot see them. The benchmark scores agents with 12,261 weighted checks that read only the observable artifacts the agent leaves behind, rewarding timely action, correct silence, and cross-stage constraint-holding. Seven frontier models were run on every task; the best scored only 32.5 on average (max@3 41.2), and all models' check pass rates decayed 10-15 points from the first to the last third of a task. The paper's central claim is that contemporary agents are far from being trustworthy proactive, persistent life assistants, and that this is now directly measurable.

What carries the argument

The load-bearing mechanism is the mutation: a background change to the world state that produces no agent turn and no notification, so the only way to notice it is to re-query the world unprompted. Around this, the benchmark organizes each task as a scripted timeline of user messages, world observations, notifications, and mutations, and grades the agent with stage-aware weighted checks in three tiers (per-stage, cross-stage, and final) that read only observable artifacts: backend state, workspace files, sent email, and reply text. The weight distribution is deliberately uneven, so a single safety or budget violation outweighs many routine sub-tasks, and high-weight checks usually require a durable artifact rather than a transient chat reply.

What would settle it

Re-run the benchmark with an agent that is told every silent change the moment it happens; if that agent also scores far below 100, the checks may be impossible or false-negative, and the measured gap would not be purely a capability gap.

Watch

Extended reading notes

Core claim

The central claim is that no existing benchmark measures what everyday life assistance actually requires, and VibeLifeBench is a working instrument for doing so. A competent life assistant must act when action is due, stay silent when it is not, notice changes that nobody announced, and keep a single plan coherent across weeks; VibeLifeBench encodes exactly those demands as weighted checks over the state of a simulated world. Evaluated on the full suite, seven frontier models all score low, with the strongest at an avg@3 of 32.5 and a best-of-three ceiling of 41.2, and the lowest capability axes are proactivity, persistence, and propagation of silent mutations. The same models show a consistent decline in per-stage pass rate as the timeline advances, which the paper reads as direct evidence that long-horizon coherence and self-consistency are precisely where current agents are weakest.

Load-bearing premise

The empirical conclusion rests on the assumption that the benchmark authors' 12,261 weighted checks reward all and only the competent behaviors, with no false negatives, and that all 200 tasks are actually solvable through the provided tool interfaces.

Editorial extensions

If this is right

  • An agent that only answers the prompt in front of it will systematically miss silent mutations, so deployed life-assistant products need scheduled re-inspection loops rather than pure instruction following.
  • Cross-stage bookkeeping in durable artifacts is the largest identified source of failure, so persistent state management is a prerequisite for trustworthy long-horizon assistance.
  • Per-stage pass rates fall by 10-15 points from the first to the last third of a task for every evaluated model, meaning output quality decays with horizon regardless of task length.
  • The cross-stage and final tiers carry disproportionate weight and the lowest pass rates, so safety and budget-cap failures are not hidden behind routine task completion.
  • No model is competent across all ten life domains, and the relative difficulty of domains is consistent across models, making the suite usable as a training target and a progress instrument.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the check suite is valid, the same living-world design could be ported to other high-stakes long-horizon domains such as health, legal, or finance, by swapping the mock services and re-authoring weighted checks.
  • The benchmark rewards calibrated initiative, not just activity: an agent that constantly re-queries and acts could pass tool-call checks but fail the staying-silent checks, so it implicitly measures judgment about when not to act.
  • A natural next experiment is to give each model a perfect summary of all prior world state at every stage; if scores stay near 32.5, the bottleneck is decision policy rather than memory, and if they jump, the gap is partly a context-management limitation.
  • The narrow spread between the best and worst model suggests that simply scaling models or improving single-turn tool use will not close the gap; a testable prediction is that training on long-horizon proactive tasks is required.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces VibeLifeBench, a benchmark of 200 long-horizon, multi-week tasks across ten everyday-life domains, simulated in a deterministic world of 22 mock services exposing 288 tool interfaces. Tasks are driven by scripted timelines of user messages, world observations, notifications, and silent mutations, and are graded by 12,261 weighted deterministic checks over observable artifacts. The authors evaluate seven frontier LLM agents under a uniform harness and report uniformly low scores (avg@3 from 21.1 to 32.5), concluding that current agents lack proactivity, persistence, and long-horizon coherence. The benchmark and evaluation framework are promised for open release.

Significance. If the instrument is valid, VibeLifeBench would be a useful addition: it operationalizes proactivity and silent mutation in a reproducible, offline, deterministic environment, with stage-aware, cross-stage, and final scoring tiers and weighted safety checks. The design has clear strengths: scoring reads observable artifacts rather than hidden reasoning, the event model cleanly separates environment-driven from user-prompted turns, and per-task variance and token costs are reported. The main weakness is that the paper does not validate the scoring apparatus. With no oracle run, no random or human baseline, and no independent audit of the check predicates, the low model scores cannot yet be attributed to a capability gap. The benchmark contribution is promising, but its central empirical claim is currently uncalibrated.

major comments (3)
  1. [Sections 2.2.5, 2.3, 4.1] The headline conclusion in Section 4.1 ('contemporary agents are still far from able...') requires that the 12,261 weighted checks are satisfiable by a competent agent and free of systematic false negatives. The paper provides no oracle run, no random baseline, no human baseline, and no independent audit of the check predicates. Table 3 includes 'Tool call' as a scored dimension, and Appendix A prescribes a single 'what a competent assistant should do' per event, so an agent that reaches the same end state through a different legitimate service, tool order, or timing can be scored as failing. Section 2.3 states that the pipeline 'avoids false negatives' but supplies no measurement. As a result, the observed 21-33 avg@3 band is equally consistent with a genuine capability gap and with checkers that only accept the canonical solution path. A minimal fix is to add a hand-authored gold-trajectory (oracle) run that should score very high, a random or trivial baseline that should score very low, and a sample-based false-negative audit; report the resulting scores and any check revisions.
  2. [Section 5.1] The capability-axis attribution used in Section 5.2 is assigned by keyword matching over check names, which the paper itself concedes is 'indicative rather than exact.' The subsequent conclusion that proactivity and persistence are the largest weaknesses leans on this mapping, yet 41.8% of failures fall outside the named categories and no validation of the axis assignment is reported. As written, this part of the failure-mode analysis is not load-bearing; please either validate the mapping (for example, by human annotation with inter-annotator agreement, or by defining the axes explicitly in the check schema) or present the axis results as exploratory rather than as evidence for the mechanism.
  3. [Sections 2.2.1 and 3.1] There is a discrepancy in the scoring protocol: the formal task definition in Section 2.2.1 says scoring criteria are evaluated when the run ends, while Section 3.1 says 'once a stage's events are processed, the scoring criteria run once against the current world state.' These are materially different protocols for per-stage checks and timeliness credit. Please state explicitly when each tier is evaluated, whether a check failed at its stage can be recovered later, and how partial credit is awarded across stages.
minor comments (6)
  1. [Section 1] The abstract and Section 1 state 'No current benchmark measures this,' but Section 6 cites UniClawBench, which is described as a 'universal benchmark for proactive agents,' and ClawMark, a 'living-world benchmark' for multi-day agents. Please qualify the novelty claim by specifying the combination of properties (life domain plus proactivity plus living world plus long horizon) that is absent from prior work.
  2. [Section 3.2] Aggregate scores are reported as averages across 200 tasks with no confidence interval or standard error; given that each task is run only three times, please report task-level uncertainty (for example, bootstrap confidence intervals) or at least the distribution of per-task scores.
  3. [Section 5.2] The Spearman correlations with number of events, horizon, and number of stages do not state the unit of analysis (tasks? model-averaged scores?) or significance levels; as written they do not support the sentence that difficulty is driven mainly by sustaining staged constraints rather than by tasks being longer.
  4. [Appendix A, Table 8] In Table 8, the stage/date label 'D1, 4/18' appears twice for two different events; please correct the duplicate date or clarify the intended timeline.
  5. [Table 1] The caption's symbol legend is not rendered legibly in the provided text; please ensure the legend appears correctly in the published version.
  6. [Section 2.4] The paper says it 'will open-source all tasks, environments, and the evaluation framework,' but no release URL or artifact is provided; for reproducibility, please include a link or state an embargo date if applicable.

Circularity Check

0 steps flagged · score 2.0 of 10

No derivation-level circularity; model scores are measurements against an author-defined benchmark, with construct-validity caveats rather than circular reasoning.

full rationale

The paper contains no fitted parameters renamed as predictions, no imported uniqueness theorem, and no ansatz smuggled in by self-citation. The formal pipeline (Eq. 1–2) defines a score as the weighted fraction of deterministic checks passed, and the reported model scores are computed measurements of that score; the claim that models are 'far from able' is a measurement against the benchmark's operational definition, not a prediction derived from the same data. No equation reduces to its own input. The genuine weaknesses are construct-validity concerns, not circularity: no oracle run, human baseline, or random baseline is reported to show the 12,261 checks are satisfiable and free of false negatives (Sections 2.2.5, 2.3); Table 3's 'Tool call' evidence dimension and Appendix A's single 'what a competent assistant should do' per event risk penalizing legitimate alternative solution paths; and Section 5.1 explicitly concedes that the capability-axis attribution is 'keyword matching... indicative rather than exact,' so the proactivity/persistence failure labels are disclaimed, not hidden. These caveats weaken the external interpretability of 'agents lack proactivity and persistence' but do not constitute a circular derivation. The one self-related citation (Terrarium [16]) is infrastructure only and is not load-bearing for the central empirical claim. Under the requirement to exhibit a specific reduction, no circular step is established.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The benchmark's central results rest on the authors' authored scoring criteria and simulation design rather than on derived mathematics. The main ledger items are the 12,261 hand-assigned check weights and the unvalidated assumption that the checks correctly encode competence. No hypothetical entities are invented, and no numeric constants are fitted to model outputs.

free parameters (1)
  • Per-check weights w_i = Not reported; manually authored
    Each task assigns 12,261 weighted checks with deliberately uneven weights (Section 2.2.5). The weights determine aggregate scores and model ordering, but no procedure, distribution, or audit is given, so scores are partly an artifact of these hand-set choices.
assumptions (4)
  • ad hoc to paper The 12,261 deterministic check predicates correctly encode competent life assistance and are free of false negatives and exploitable holes.
    Section 2.3 asserts checks are discriminating and discoverable, but no oracle run, human baseline, or independent audit validates the checks.
  • domain assumption The observable simulator state and agent artifacts capture all behavior that matters for scoring.
    Section 2.2.5 restricts scoring to observable artifacts and never reads hidden reasoning; this assumes invisible reasoning has no effect on outcomes.
  • domain assumption Running each model under the openclaw harness at its strongest reasoning setting isolates model capability from scaffold effects.
    Section 3.3 states all models share a scaffold, but no scaffold ablation or alternative harness results are reported.
  • domain assumption Mutations are genuinely silent and visible only via proactive re-inspection.
    Section 2.2.3 defines mutations as producing no turn; the proactivity measurement inherits this design assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?." pith.science (2026). https://pith.science/paper/PFOPO3BY

@misc{pith2026260810875,
  author       = {Pith},
  title        = {Pith review of: VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PFOPO3BY}},
  note         = {Machine review of arXiv:2608.10875}
}
read the original abstract

Large language model (LLM) agents are increasingly deployed as personal assistants. Existing evaluations, however, mostly use short, self-contained requests in static environments. Everyday life assistance is different. A task runs for weeks rather than minutes. The world keeps changing while the agent is not being prompted. Many constraints are never stated outright. An agent that merely answers the request in front of it will fail at such a task. What is needed instead is an agent that stays proactive and consistent. It decides on its own when to act, when to ask, and when to stay silent. It notices changes that nobody announced. It keeps one plan coherent from the first day to the last. No current benchmark measures this. We introduce VibeLifeBench, a benchmark of 200 long-horizon tasks across ten everyday-life domains. Each task is a scripted multi-week timeline in a simulated world of 22 mock services. The world advances on its own clock, and many of its changes are silent, so only an agent that re-inspects the world discovers them. Every task is graded by fine-grained, weighted checks that read only what the agent actually left behind, covering the end state, the timeliness of its actions, and whether it upheld the implicit constraints. We evaluate seven frontier models. All of them score low, which shows how far current agents are from assisting with real life. We will open-source all tasks, environments, and the evaluation framework.

Figures

Figures reproduced from arXiv: 2608.10875 by the authors.

Figure 1
Figure 1. Overview of VibeLifeBench. The top shows a complete living-world task end to end: a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Number of tasks that use each service across the suite (all 22 services are exercised). Usage is [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Per-domain distribution of horizon, number of events, number of services, and number of [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Per-run input tokens, output tokens, tool calls, and turns. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Per-stage check pass rate as a function of the normalized position of the check along the task [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 4 canonical work pages

  1. [1]

    Swe-milestone: Evaluating ai agents on continuous software evolution, 2026

    Gangda Deng, Zhaoling Chen, Zhongming Yu, Haoyang Fan, Yuhong Liu, Yuxin Yang, Dhruv Parikh, Rajgopal Kannan, Le Cong, Mengdi Wang, Qian Zhang, Viktor Prasanna, Xiangru Tang, and Xingyao Wang. Swe-milestone: Evaluating ai agents on continuous software evolution, 2026. URL https://arxiv.org/abs/2603.13428

    arXiv 2026

  2. [2]

    Merrill, Alexander G

    Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, An...

    arXiv 2026

  3. [3]

    Apex-agents, 2026

    Bertie Vidgen, Austin Mann, Abby Fennelly, John Wright Stanly, Lucas Rothman, Marco Burstein, Julien Benchek, David Ostrofsky, Anirudh Ravichandran, Debnil Sur, Neel Venugopal, Alannah Hsia, Isaac Robinson, Calix Huang, Olivia Varones, Daniyal Khan, Michael Haines, Austin Bridges, Jesse Boyle, Koby Twist, Zach Richards, Chirag Mahapatra, Brendan Foody, an...

    arXiv 2026

  4. [4]

    Edgebench: Unveiling scaling laws of learning from real-world environments, 2026

    Deyao Zhu, Xin Zhou, Shengling Qin, Xuekai Zhu, Hangliang Ding, Shu Zhong, Zixin Wen, Zhonglin Xie, Chenhui Gou, Linxuan Ren, Yueyang Wang, Junfeng Zhong, Rui Liu, Tian Gao, Yangguang Lin, Jingyuan Zhang, Maojia Song, Xuan Qi, Jinhong Wu, Chenyang Zhang, Yinzhu Piao, Ziru Niu, Hongbin Lin, Lingxiang Meng, Peng Tang, Chengyao Tang, Shanyu Wu, Huanyu Zheng,...

    arXiv 2026

  5. [5]

    Glaze, Gabriel Orlanski, Ramya Ramakrishnan, Benji Xu, Asim Biswal, Vincent Sunn Chen, Frederic Sala, Matei Zaharia, and Joseph E

    Parth Asawa, Christopher M. Glaze, Gabriel Orlanski, Ramya Ramakrishnan, Benji Xu, Asim Biswal, Vincent Sunn Chen, Frederic Sala, Matei Zaharia, and Joseph E. Gonzalez. Continual learning bench: Evaluating frontier ai systems in real-world stateful environments, 2026. URL https://arxiv.org/abs/2606.05661

    arXiv 2026

  6. [6]

    Jobbench: Aligning agent work with human will, 2026

    Yuetai Li, Yichen Feng, Zhangchen Xu, Zixian Ma, Kaiyuan Zheng, Fengqing Jiang, Xinghua Sun, Rulin Shao, Zichen Chen, Yue Huang, Xinyang Han, Brian Lee, Kayla Xu, Shenglai Zeng, Hang Hua, Xiangliang Zhang, Basel Alomair, Ranjay Krishna, Luke Zettlemoyer, Pang Wei Koh, Bhaskar Ramasubramanian, Luyao Niu, Xiang Yue, and Radha Poovendran. Jobbench: Aligning ...

    arXiv 2026

  7. [7]

    Workspace-bench 1.0: Benchmarking ai agents on workspace tasks with large-scale file dependencies,

    Zirui Tang, Xuanhe Zhou, Yumou Liu, Linchun Li, Yukai Wu, Weizheng Wang, Hongzhang Huang, Wei Zhou, Jun Zhou, Jiachen Song, Shaoli Yu, Jinqi Wang, Zihang Zhou, Hongyi Zhou, Yuting Lv, Jinyang Li, Jiashuo Liu, Ruoyu Chen, Chunwei Liu, GuoLiang Li, Jihua Kang, and Fan Wu. Workspace-bench 1.0: Benchmarking ai agents on workspace tasks with large-scale file d...

    work page

  8. [8]

    Fung, Chun Yuan, and Li Shen

    Haotian Luo, Huaisong Zhang, Xuelin Zhang, Haoyu Wang, Zeyu Qin, Wenjie Lu, Guozheng Ma, Haiying He, Yingsha Xie, Qiyang Zhou, Zixuan Hu, Hongze Mi, Yibo Wang, Naiqiang Tan, Hong Chen, Yi R. Fung, Chun Yuan, and Li Shen. Ultrahorizon: Benchmarking agent capabilities in ultra long-horizon scenarios, 2025. URLhttps://arxiv.org/abs/2509.21766

    arXiv 2025

Show all 26 references
  1. [9]

    Yuxuan Zhang, Yubo Wang, Yipeng Zhu, Penghui Du, Junwen Miao, Xuan Lu, Zhuofeng Li, Xingwei Qu, Zhengkang Guo, Yuanzhe Shen, Dingjie Song, Han Zhou, Tuney Zheng, Xian Wu, Hao Yu, Songcheng Cai, Yi Lu, Yunzhuo Hao, Minyi Lei, Liang Chen, Kai Zou, Huifeng Yin, Wendong Xu, Dongfu...

    2026 arXiv

  2. [10]

    Uniclawbench: A universal benchmark for proactive agents on real-world tasks, 2026

    Zhekai Chen, Chengqi Duan, Kaiyue Sun, Bohao Li, Yuqing Wang, Manyuan Zhang, and Xihui Liu. Uniclawbench: A universal benchmark for proactive agents on real-world tasks, 2026. URL https://arxiv.org/abs/2607.08768

    2026 arXiv

  3. [11]

    Claw-eval: Towards trustworthy evaluation of autonomous agents, 2026

    Bowen Ye, Rang Li, Qibin Yang, Yuanxin Liu, Linli Yao, Hanglong Lv, Zhihui Xie, Chenxin An, Lei Li, Lingpeng Kong, Qi Liu, Zhifang Sui, and Tong Yang. Claw-eval: Towards trustworthy evaluation of autonomous agents, 2026. URLhttps://arxiv.org/abs/2604.06132

    2026 arXiv

  4. [12]

    Wildclawbench: A benchmark for real-world, long-horizon agent evaluation, 2026

    Shuangrui Ding, Xuanlang Dai, Long Xing, Shengyuan Ding, Ziyu Liu, Yang JingYi, Penghui Yang, Zhixiong Zhang, Xilin Wei, Xinyu Fang, Yubo Ma, Haodong Duan, Jing Shao, Jiaqi Wang, Dahua Lin, Kai Chen, and Yuhang Zang. Wildclawbench: A benchmark for real-world, long-horizon agen...

    2026 arXiv

  5. [13]

    Jiayu Liu, Cheng Qian, Zhaochen Su, Qing Zong, Shijue Huang, Bingxiang He, and Yi R. Fung. Costbench: Evaluating multi-turn cost-optimal planning and adaptation in dynamic environments for llm tool-use agents, 2026. URLhttps://arxiv.org/abs/2511.02734

    2026 arXiv

  6. [14]

    Clawmark: A living-world benchmark for multi-turn, multi-day, multimodal coworker agents, 2026

    Fanqing Meng, Lingxiao Du, Zijian Wu, Guanzheng Chen, Xiangyan Liu, Jiaqi Liao, Chonghe Jiang, Zhenglin Wan, Jiawei Gu, Pengfei Zhou, Rui Huang, Ziqi Zhao, Shengyuan Ding, Ailing Yu, Bo Peng, Bowei Xia, Hao Sun, Haotian Liang, Ji Xie, Jiajun Chen, Jiajun Song, Liu Yang, Ming X...

    2026 arXiv

  7. [15]

    Clawarena: Benchmarking ai agents in evolving information environments, 2026

    Haonian Ji, Kaiwen Xiong, Siwei Han, Peng Xia, Shi Qiu, Yiyang Zhou, Jiaqi Liu, Jinlong Li, Bingzhou Li, Zeyu Zheng, Cihang Xie, and Huaxiu Yao. Clawarena: Benchmarking ai agents in evolving information environments, 2026. URLhttps://arxiv.org/abs/2604.04202

    2026 arXiv

  8. [16]

    Terrarium: Multi-turn data engine for evaluating and optimizing llm agents in living environments

    Evolvent AI. Terrarium: Multi-turn data engine for evaluating and optimizing llm agents in living environments. https://github.com/evolvent-ai/Terrarium, 2026. Open-source evaluation infrastructure

    2026

  9. [17]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL https://arxiv.org/abs/2310.06770

    2024 arXiv

  10. [18]

    Swt-bench: Testing and validating real-world bug-fixes with code agents, 2025

    Niels Mündler, Mark Niklas Müller, Jingxuan He, and Martin Vechev. Swt-bench: Testing and validating real-world bug-fixes with code agents, 2025. URLhttps://arxiv.org/abs/2406.12952

    2025 arXiv

  11. [19]

    Deepswe: Measuring frontier coding agents on original, long-horizon engineering tasks, 2026

    Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. Deepswe: Measuring frontier coding agents on original, long-horizon engineering tasks, 2026. URL https://arxiv.org/abs/2607.07946

    2026 arXiv

  12. [20]

    Spreadsheetbench: Towards challenging real world spreadsheet manipulation, 2024

    Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, and Jie Tang. Spreadsheetbench: Towards challenging real world spreadsheet manipulation, 2024. URLhttps://arxiv.org/abs/2406.14991

    2024 arXiv

  13. [21]

    Pptbench: Towards holistic evaluation of large language models for powerpoint layout and design understanding, 2025

    Zheng Huang, Xukai Liu, Tianyu Hu, Kai Zhang, and Ye Liu. Pptbench: Towards holistic evaluation of large language models for powerpoint layout and design understanding, 2025. URL https: //arxiv.org/abs/2512.02624

    2025

  14. [22]

    Kim, Patrick Chao, Samuel Miserendino, Gildas Chabot, David Li, Michael Sharman, Alexandra Barr, Amelia Glaese, and Jerry Tworek

    Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Simón Posada Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, Natalie S. Kim, Patrick Chao, Samuel Miserendino, Gildas Chabot, David Li, Michael Sharman, Alexandra Barr, Ameli...

    2025 arXiv

  15. [23]

    Aa-briefcase, 2026

    Artificial Analysis. Aa-briefcase, 2026. URL https://huggingface.co/datasets/ ArtificialAnalysis/AA-Briefcase-Lite

    2026

  16. [24]

    Lifelongagentbench: Evaluating llm agents as lifelong learners, 2025

    Junhao Zheng, Xidi Cai, Qiuke Li, Duzhen Zhang, ZhongZhi Li, Yingying Zhang, Le Song, and Qianli Ma. Lifelongagentbench: Evaluating llm agents as lifelong learners, 2025. URL https: //arxiv.org/abs/2505.11942

    2025 arXiv

  17. [25]

    R-horizon: How far can your large reasoning model really go in breadth and depth?, 2025

    Yi Lu, Jianing Wang, Linsen Guo, Wei He, Hongyin Tang, Tao Gui, Xuanjing Huang, Xuezhi Cao, Wei Wang, and Xunliang Cai. R-horizon: How far can your large reasoning model really go in breadth and depth?, 2025. URLhttps://arxiv.org/abs/2510.08189. 16 A A Complete Task Walkthroug...

    2025

  18. [2026]

    URLhttps://arxiv.org/abs/2605.03596

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.