Artificial Intelligence · arXiv:2611.25146 v1 · submitted 18 Nov 2026 · 32 references
EverydayBench: Evaluating Satisfactory Completion of Live User-Facing Digital Work
The paper says
“Everyday digital tasks, from selecting products and planning trips to conducting research and creating artifacts, require AI agents to deliver useful outcomes rather than merely retrieve web information.”
more of the abstract
Yet evaluating these outcomes on the live web is challenging, as a single request may admit multiple valid results, and changing web information can alter which results remain valid over time. Although existing benchmarks address aspects of this setting, they do not fully integrate predefined success criteria, current evidence verification, and deliverable-quality assessment for agent-selected outcomes. To address this gap, we introduce EverydayBench, a benchmark of 77 human-authored tasks spanning eight domains of everyday life and work. Each task specifies success criteria before execution without prescribing an answer or workflow, allowing agents to select their own outcomes. To evaluate submissions under fixed success criteria, EverydayBench combines (1) deterministic checks of explicit constraints, (2) verification of claims against agent-cited webpages retrieved at evaluation time, and (3) rubric-based assessment of final deliverable quality. We further validate the protocol through human judgments, repeated scoring under fixed evidence, and controlled grounding failures. Across 23 reported agent-model configurations, Claude Opus 5.5 and Claude Fable 5.1 tie for the highest strict pass rate at 54.55%; a controlled GPT-6 Sol effort sweep ranges from 29.87% at low effort to 45.45% at max effort. Reliability varies substantially by task type: GPT-5.6 Sol leads lookup, Fable 5.1 leads planning, Opus 5.5 and Fable 5.1 tie on comparison, and GPT-6 Astra and Sonnet 5.5 tie on customized deliverables. Together, these findings highlight the difficulty of delivering complete, well-supported, and usable outcomes for everyday user requests.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 20 Nov 2026 · read the full text (120,117 characters) · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. EverydayBench is a well-validated benchmark for evaluating whether AI agents deliver satisfactory outcomes on live-web everyday tasks, with a careful evaluation methodology that combines deterministic checks, source-grounded verification, and rubric-based scoring. This line is wrong
8 minor
How often objections from reviews like this one are right: not yet measured
The paper, first character to last
The reviewer was given the whole text, 120,117 characters. Not checked: figures, code, what the 32 references say.
8 minor objections
The paper says, at Abstract
Pith says: minor objection 1 of 8
The abstract mentions '23 reported agent-model configurations' but the main results table (Table 1) shows 23 configurations including the effort sweep. The count is correct but the phrasing could be clearer about which configurations are in the primary leaderboard vs. the effort sweep.
The paper says, at §3.2, Evaluation Setup
Pith says: minor objection 2 of 8
The paper states 'The judging backbone is GPT-5.4 at temperature zero' but does not discuss potential biases of using an LLM as a judge, even though it validates against human judgments. A brief discussion of the limitations of LLM-as-a-judge would strengthen the paper.
show 6 more
The paper says, at §4.2, Table 1
Pith says: minor objection 3 of 8
The table caption says 'Bold marks the best pass rate in each column' but the bold entries in the 'Plan' column show 66.67 for Fable 5.1 and 58.33 for several others, which is correct. However, in the 'Build' column, GPT-6 Astra and Sonnet 5.5 both show 76.92 but neither is bolded. The caption says bold marks the best, so both should be bolded.
The paper says, at §4.3, Failure Mode Analysis
Pith says: minor objection 4 of 8
The failure mode analysis is thorough but the description of the failure-mode judge's agreement with human labels (92 of 100) could be reported as a percentage (92%) for consistency with other agreement metrics in the paper.
The paper says, at §5, Appendix A.1
Pith says: minor objection 5 of 8
The pass-threshold sensitivity analysis is valuable but the table would benefit from a note explaining why the 19 configurations in the primary leaderboard exclude the 4 effort-sweep configurations. This is clear from context but could be stated explicitly.
The paper says, at §5, Appendix A.2
Pith says: minor objection 6 of 8
The computer-use vs. retrieval-only comparison on 25 tasks is interesting but the sample size is small. The paper should note that the 25 tasks were sampled and whether they are representative of the full 77-task set.
The paper says, at §5, Appendix B
Pith says: minor objection 7 of 8
The human validation section reports Cohen's kappa values but does not specify whether the kappa values are for exact agreement or weighted agreement. The main text says 'linearly weighted Cohen's κ' but the appendix should also state this clearly.
The paper says, at §5, Appendix C
Pith says: minor objection 8 of 8
The failure mode analysis appendix is very detailed and well-structured. However, the prompts for the failure-mode judge (Prompts 1-3b) are very long and complex. A brief summary of how the prompts were developed and validated would help readers assess the reliability of the failure mode classification.
What would settle it
No significant objection identified
from the skeptic's reading
How the review went
5 passes by deepseek-v4-flash · 20 Nov 2026
- Reader holds up
- Skeptic agrees
- Referee 8 minor
- Check against the paper
- Editor wrote the verdict above
Not run on this paper: the simulated authors' defence.
Pith checked this review's objections against the text of the paper it read.
The review in full
The review in brief, machine-written
EverydayBench addresses a real gap: most agent benchmarks evaluate whether an agent can retrieve information or perform actions, but not whether the final deliverable is actually useful for the person who asked. The paper's central design choice—fix the standard of completion, not the solution—is sensible, and the three-part evaluation (direct checks, source grounding, rubric scoring) is a genuine methodological contribution.
The validation is thorough. Human agreement studies for both grounding (κ=0.695) and rubric judging (κ=0.644) are reported, temporal stability is examined, and the seeded-fault detection rate of 90.7% gives confidence that the evaluator catches real failures. The failure-mode analysis is particularly valuable: it shows concretely where agents lose points, and the finding that even top configurations lose most points because "a claim says more than the page it cites" is both specific and actionable.
The main limitation is inherent to the approach: 77 tasks, however carefully curated, cannot cover the full space of everyday digital work. The authors acknowledge this. A second concern is that the LLM-as-a-judge component, while validated against humans, still relies on GPT-5.4 as the backbone; the paper's own ablation shows that the full evaluator is more stable than simpler baselines, but the judge's reliability on tasks far from its training distribution is untested. These are minor caveats for a benchmark paper.
This paper is for anyone building or evaluating web agents who wants to move beyond retrieval accuracy toward outcome quality. It deserves a serious referee and would benefit from community adoption and extension.
Recommendation: accept with minor revisions. The methodology is sound, the validation is adequate, and the benchmark fills a clear gap.
The strongest objection
No significant objection identified
No significant objection identified
The referee's report
The benchmark addresses a real gap in evaluating AI agents for everyday digital work on the live web, where multiple valid outcomes exist and web content changes over time. The evaluation methodology combining deterministic checks, source verification, and human-validated rubrics is well-designed and validated. The failure mode analysis provides actionable insights for improving agent systems. The benchmark is likely to be useful for the community.
Pith's reading of the paper (a machine's summary, not the authors')
The paper argues that evaluating AI agents on everyday digital work should be judged by whether the delivered outcome is satisfactory — it meets the stated requirements, is usable for its intended purpose, and is supported by current web evidence — rather than by whether it matches a reference answer. To make that standard measurable, it introduces EverydayBench, a benchmark of 77 human-authored tasks across eight domains, where success criteria are fixed before execution but agents are free to choose their own sources, workflows, and concrete results. Each submission is scored by combining deterministic checks of explicit constraints, verification of claims against the live pages the agent cites, and rubric-based scoring of the final deliverable's quality, an assessment validated against independent human judgments. Across 23 agent–model configurations, the best strict pass rates tie at 54.55%, and only 32.30% of all runs meet the benchmark's completion standard, evidence that producing plausible-looking information and delivering complete, usable, supported work are different things.
What carries the argument. The load-bearing mechanism is the three-route criterion verification over an outcome schema. Each of a task's success criteria (7.6 per request on average) is assigned to one of three routes: deterministic direct tests for properties checkable from the submission alone (arithmetic, cross-field consistency, artifact structure); source-grounded checks that determine whether a submitted factual claim is supported by the exact agent-cited page retrieved at evaluation time; and rubric-based component scoring in which each predefined outcome component is judged on a 1–5 scale against human-curated anchors for scores 1, 3, and 5, with a format-violating output scoring 0 on every component. Component scores are weight-averaged and scaled to $[0,1]$, and a task passes only when its overall score is at least 85% and no component scores below 3.
Core claim. EverydayBench's central claim is that satisfactory task completion can be operationalized as an outcome-level standard: fix the completion criteria before execution while leaving the solution open, then assess the agent-selected outcome through three complementary routes — deterministic checks of explicit constraints, verification of factual claims against the exact webpages the agent cites retrieved at evaluation time (with an inaccessible page yielding an unverified finding rather than a factual rejection), and rubric-based component scoring with human-curated 1/3/5 anchors. Under this protocol, judged by GPT-5.4 at temperature zero, the strongest agents (Claude Code with Opus 5.5 and with Fable 5.1) tie for first at a 54.55% strict pass rate, and only 32.30% of 1,771 agent–task runs pass. The failure analysis shows that the most common way even frontier agents lose points is a claim that says more than the page it cites, while lower-cost agents lose points mainly through output files that are missing or violate the required format and through user needs the model never investigates.
The premise it rests on. The load-bearing premise is that the 77 human-authored tasks are representative enough of the broad space of everyday digital work, and that the evaluation protocol — deterministic checks, source-grounding against live cited pages, and rubric scoring validated against human judges — measures satisfactory completion validly and reliably for those tasks.
If the paper is right (Pith's inferences, not the authors')
If the benchmark is valid, it gives the field a yardstick for everyday agent work that survives web change: because claims are checked against live pages at evaluation time, a score reflects whether the outcome is still valid when judged, not only when produced.
The reported validation numbers — mean human–evaluator agreement of $\kappa=0.695$ on grounding and $\kappa=0.644$ on rubric scoring, close to or above human–human agreement — support treating the judge's scores as reproducible for benchmarking comparisons.
The result that no configuration leads across all five task types (lookup, comparison, planning, analysis, build) implies systems should be selected by the mix of work an application emphasizes, not by aggregate ranking alone.
The system-wide 32.30% pass rate sets a concrete target: current agents complete everyday delegated work at most about half the time even at the frontier, and additional reasoning effort raises the upper envelope of reliability rather than monotonically guaranteeing more completed tasks.
The finding that harness choice with a fixed model changes pass rate by 8–20 points implies agentic infrastructure, not only the underlying model, is a lever on satisfactory completion.
A natural extension the authors do not pursue: whether the same three-route protocol transfers to safety- or compliance-critical delegation, where an unacted-upon implicit need (like the short-term-rental ordinance example) carries higher stakes than in shopping or trip planning.
The paper's own failure taxonomy suggests a testable hypothesis beyond its scope: the dominant frontier failure, 'claim says more than the cited page,' is concentrated in how claims are phrased in the final deliverable rather than in the research itself, which would imply that grounding-aware generation or post-hoc claim-verification of deliverables could recover many lost points.
Because the assessor is a single frozen judge (GPT-5.4), the reported 54.55% is one judge's reading; a robustness check not done in the paper would be scoring the same 1,771 runs with a judge of a different model family and with an independent human panel to see whether the leaderboard ordering and the 32.30% aggregate survive.
Check this review
What was read
The passes were given all 120,117 characters of the paper's LaTeX source.
Fingerprint of that text (SHA-256): 6fb9046be597b35164159190a8efa8179294251e0665bb6e57c04d39b0364fb1
Pith does not keep that text. Rebuilt from the paper's source or PDF by the same code, it has this fingerprint if it is the same text. From a copy of Pith's code:
PYTHONPATH=. python scripts/review_input_check.py 2611.25146
The passes, in the order they ran
- Reader deepseek-v4-flash · medium effort · $0.0056 · instructions of the Reader, fingerprint f9ead7300382…
- Skeptic deepseek-v4-flash · medium effort · $0.0031 · instructions of the Skeptic, fingerprint 8f88c0f61488…
- Referee deepseek-v4-flash · medium effort · $0.0034 · instructions of the Referee, fingerprint a852d99aea0d…
- Editor deepseek-v4-flash · medium effort · $0.0033 · instructions of the Editor, fingerprint e7211f6a3d27…
- Summary for the page deepseek-v4-flash · medium effort · temperature 0.2 · $0.0066 · instructions of the Summary for the page, fingerprint e4f56bf568e7…
The passes cost $0.022 in model calls together.
Version stamped on this review: pith-pipeline@v1.4.1-daily-deepseek-or-medium+deepseek/deepseek-v4-flash+p.448e4af8143a
Each link opens a pass's instructions as plain text, the exact bytes its fingerprint was taken over. Every stored set of instructions.
curl -s https://pith.science/methodology/prompts/f9ead730038216984cfb56d7cb34332bdf691828296a1de4245f3d6667d4bf56 | sha256sum
prints the fingerprint of the reader's instructions. The same works for every pass.
The receipt
The review's fingerprint was taken on 20 Nov 2026. The review you are reading still matches it.
Fingerprint of the review (SHA-256): ecafe8e23ba4d51b20ca87cf1ae884d2b2456fdaa691d12af0c982e68fcd9de4
curl -s 'https://pith.science/api/v1/verdicts/2611.25146?form=canonical' | sha256sum
prints the fingerprint of the review as it stands now. The rule, in four steps.
A matching fingerprint shows the review has not changed since that date. It does not show that the review is right.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, from the paper's LaTeX source.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Everyday digital tasks, from selecting products and planning trips to conducting research and creating artifacts, require AI agents to deliver useful outcomes rather than merely retrieve web information. Yet evaluating these outcomes on the live web is challenging, as a single request may admit multiple valid results, and changing web information can alter which results remain valid over time. Although existing benchmarks address aspects of this setting, they do not fully integrate predefined success criteria, current evidence verification, and deliverable-quality assessment for agent-selected outcomes. To address this gap, we introduce EverydayBench, a benchmark of 77 human-authored tasks spanning eight domains of everyday life and work. Each task specifies success criteria before execution without prescribing an answer or workflow, allowing agents to select their own outcomes. To evaluate submissions under fixed success criteria, EverydayBench combines (1) deterministic checks of explicit constraints, (2) verification of claims against agent-cited webpages retrieved at evaluation time, and (3) rubric-based assessment of final deliverable quality. We further validate the protocol through human judgments, repeated scoring under fixed evidence, and controlled grounding failures. Across 23 reported agent-model configurations, Claude Opus 5.5 and Claude Fable 5.1 tie for the highest strict pass rate at 54.55%; a controlled GPT-6 Sol effort sweep ranges from 29.87% at low effort to 45.45% at max effort. Reliability varies substantially by task type: GPT-5.6 Sol leads lookup, Fable 5.1 leads planning, Opus 5.5 and Fable 5.1 tie on comparison, and GPT-6 Astra and Sonnet 5.5 tie on customized deliverables. Together, these findings highlight the difficulty of delivering complete, well-supported, and usable outcomes for everyday user requests.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-20
The authors' main claim, as the classifier picked it out 1.00
“Together, these findings highlight the difficulty of delivering complete, well-supported, and usable outcomes for everyday user requests.”
How the main claim is stated: asserted 0.97
What the abstract itself states, as probabilities: claims first 0.13 · states limitation 0.45 · human participants 0.72 · controlled comparison 0.31 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.71 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Everyday digital tasks, from selecting products and planning trips to conducting research and creating artifacts, require AI agents to deliver useful outcomes rather than merely retrieve web information. | background 1.00 | 0.06 | 0.05 | asserted 1.00 |
| Yet evaluating these outcomes on the live web is challenging, as a single request may admit multiple valid results, and changing web information can alter which results remain valid over time. | background 0.92 | 0.08 | 0.03 | asserted 0.98 |
| Although existing benchmarks address aspects of this setting, they do not fully integrate predefined success criteria, current evidence verification, and deliverable-quality assessment for agent-selected outcomes. | gap 1.00 | 0.07 | 0.05 | asserted 1.00 |
| To address this gap, we introduce EverydayBench, a benchmark of 77 human-authored tasks spanning eight domains of everyday life and work. | contribution 0.98 | 0.92 | 0.04 | asserted 1.00 |
| Each task specifies success criteria before execution without prescribing an answer or workflow, allowing agents to select their own outcomes. | method 0.79 | 0.82 | 0.03 | asserted 1.00 |
| To evaluate submissions under fixed success criteria, EverydayBench combines (1) deterministic checks of explicit constraints, (2) verification of claims against agent-cited webpages retrieved at evaluation time, and (3) rubric-based assessment of final deliverable quality. | method 0.97 | 0.90 | 0.05 | asserted 1.00 |
| We further validate the protocol through human judgments, repeated scoring under fixed evidence, and controlled grounding failures. | method 0.93 | 0.95 | 0.03 | asserted 1.00 |
| Across 23 reported agent-model configurations, Claude Opus 5.5 and Claude Fable 5.1 tie for the highest strict pass rate at 54.55%; a controlled GPT-6 Sol effort sweep ranges from 29.87% at low effort to 45.45% at max effort. | result 1.00 | 0.95 | 0.09 | asserted 1.00 |
| Reliability varies substantially by task type: GPT-5.6 Sol leads lookup, Fable 5.1 leads planning, Opus 5.5 and Fable 5.1 tie on comparison, and GPT-6 Astra and Sonnet 5.5 tie on customized deliverables. | result 1.00 | 0.94 | 0.07 | asserted 1.00 |
| Together, these findings highlight the difficulty of delivering complete, well-supported, and usable outcomes for everyday user requests. | interpretation 1.00 | 0.80 | 0.82 | asserted 0.97 |
What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-20: 1 request, 7,829 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (9)
Figures are the authors', reproduced from the paper.
References (32)
- [1] Alibaba Cloud. Model studio model inference pricing.https://www.alibabacloud.com/help/ en/model-studio/model-pricing, 2026. Accessed November 5, 2026 on Pith
- [2] Anthropic. Claude model pricing.https://www.anthropic.com/pricing#api, 2026. Accessed November 5, 2026 on Pith
- [3] R. K. Arora, J. Wei, R. S. Hicks, P. Bowman, J. Quiñonero-Candela, F. Tsimpourlas, M. Sharman, M. Shah, A. Vallone, A. Beutel, et al. HealthBench: Evaluating large language models towards improved human health.arXiv preprint arXiv:2505.08775, 2025 on PithPith's reviewarXiv
- [4] C. Bandi, R.-G. Dumitru, B. Hertzberg, D. Agarwal, G. Boo, T. Polakam, S. Hassaan, J. Da, H. Kim, V. Gupta, M. Sharma, A. Park, M. Dimakis, E. G. H. Montoya, D. Rambado, I. Salazar, R. Cruz, M.Rezaei, C.Rane, B.Levin, D.Y.Zhang, B.Kenstler, andB.Liu. Mcp-atlas: Alarge-scaleben... on Pith
- [5] J. Cohen. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychologicalbulletin, 70(4):213, 1968 on Pith
- [6] X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su. Mind2Web: Towards a generalistagentfortheweb. AdvancesinNeuralInformationProcessingSystems, 36:28091–28114, 2023 on Pith
- [7] M. Du, B. Xu, C. Zhu, L. Zhang, X. Wang, and Z. Mao. DeepResearch Bench: A comprehensive benchmark for deep research agents. InInternational Conference on Learning Representations, pages 42414–42448, 2026 on Pith
- [8] B. Gou, Z. Huang, Y. Ning, Y. Gu, M. Lin, W. Qi, A. Kopanev, B. Yu, B. Jimenez Gutierrez, Y. Shu, et al. Mind2Web 2: Evaluating agentic search with agent-as-a-judge.Advances in Neural Information Processing Systems, 38, 2026 on Pith
- [9] H. He, W. Yao, K. Ma, W. Yu, Y. Dai, H. Zhang, Z. Lan, and D. Yu. WebVoyager: Building an end-to- end web agent with large multimodal models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume1: Long Papers), pages 6864–6890, 2024 on Pith
- [10] F. Huang, S. Chbeir, A. Khatua, S. Wang, S. Tan, K. Ye, L. Bailey, M. Daniel, R. Louie, S. Koyejo, and E. Adeli. TherapyGym: Evaluating and aligning clinical fidelity and safety in therapy chatbots. In Advances in Neural InformationProcessing Systems (Evaluationsand Datasets T... on Pith
- [11] L. K. Jang, J. Y. Koh, D. Fried, and R. Salakhutdinov. Odysseys: Benchmarking web agents on realistic long horizon tasks.arXiv preprint arXiv:2604.24964, 2026 on PithPith's reviewarXiv
- [12] Z. Jiang, F. Huang, H. Xing, X. Wu, Y. Gao, R. Cao, M. Wang, S. Liu, and Y. Li. Demystifying agent skills: Why they work—until they don’t.arXiv preprint arXiv:2608.14036, 2026 on PithPith's reviewarXiv
- [13] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, pages 54107–54157, 2024 on Pith
- [14] J. Li, W. Zhao, J. Zhao, W. Zeng, H. Wu, X. Wang, R. Ge, Y. Cao, Y. Huang, W. Liu, et al. The Tool Decathlon: Benchmarking language agents for diverse, realistic, and long-horizon task execution. In International Conference on Learning Representations, pages 53359–53394, 2026 on Pith
- [15] X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, et al. AgentBench: 14 EverydayBench: Evaluating Satisfactory Completion of Live User-Facing Digital Work Evaluating LLMs as agents. InInternationalConferenceonLearning Representations, pages 52989... on Pith
- [16] Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu. G-Eval: NLG evaluation using GPT-4 with better human alignment. InProceedings of the 2023 conference on empirical methods in natural language processing, pages 2511–2522, 2023 on Pith
- [17] G. Mialon, C. Fourrier, T. Wolf, Y. LeCun, and T. Scialom. GAIA: a benchmark for general AI assistants. InInternational Conference on Learning Representations, pages 9025–9049, 2024 on Pith
- [18] S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 conference on empirical methods in natural language pro... on Pith
- [19] OpenAI. Api pricing. https://developers.openai.com/api/docs/pricing, 2026. Accessed November 5, 2026 on Pith
- [20] T. Patwardhan, R. Dias, E. Proehl, G. Kim, M. Wang, O. Watkins, S. Fishman, M. Aljubeh, P. Thacker, L. Fauconnet, et al. GDPval: Evaluating AI model performance on real-world economically valuable tasks. InInternational Conference on Learning Representations, pages 24005–24040, 2026 on Pith
show 12 more
- [21] M. Sharma, C. B. C. Zhang, C. Bandi, C. Wang, A. Aich, H. Nghiem, T. Rabbani, Y. Htet, B. Jang, S. Basu, et al. ResearchRubrics: A benchmark of prompts and rubrics for evaluating deep research agents. InInternational Conference on Learning Representations, pages 90447–90472, 2026 on Pith
- [22] G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson, et al. PaperBench: evaluating AI’s ability to replicate AI research.arXiv preprint arXiv:2504.01848, 2025 on PithPith's reviewarXiv
- [23] C. Team, S. Hao, Z. Zhang, Z. Liang, T. Liu, Y. Zha, Q. Gao, J. Chen, Z. Wang, Z. Cheng, et al. CocoaBench: Evaluating unified digital agents in the wild.arXiv preprint arXiv:2604.11201, 2026 on PithPith's reviewarXiv
- [24] J. Wang, Y. Ming, R. Dulepet, Q. Chen, A. Xu, Z. Ke, F. Sala, A. Albarghouthi, C. Xiong, and S. Joty. LiveResearchBench: A live benchmark for user-centric deep research in the wild. InInternational Conference on Learning Representations, pages 95096–95143, 2026 on Pith
- [25] J. Wei, Z. Sun, S. Papay, S. McKinney, J. Han, I. Fulford, H. W. Chung, A. T. Passos, W. Fedus, and A. Glaese. BrowseComp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516, 2025 on PithPith's reviewarXiv
- [26] T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, et al. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural InformationProcessing Systems, 37:52040–52094, 2024 on Pith
- [27] T. Xue, W. Qi, T. Shi, C. H. Song, B. Gou, D. Song, H. Sun, and Y. Su. An illusion of progress? assessing the current state of web agents.arXiv preprint arXiv:2504.01382, 2025 on PitharXiv
- [28] S.Yao,N.Shinn,P.Razavi,andK.Narasimhan. 𝜏-bench: Abenchmarkfortool-agent-userinteraction in real-world domains.arXiv preprint arXiv:2406.12045, 2024 on PithPith's reviewarXiv
- [29] O. Yoran, S. J. Amouyal, C. Malaviya, B. Bogin, O. Press, and J. Berant. AssistantBench: Can web agents solve realistic and time-consuming tasks? InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8938–8968, 2024. 15 EverydayBench: ... on Pith
- [30] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.Advances in neural information processing systems, 36:46595–46623, 2023 on Pith
- [31] S.Zhou,F.F.Xu,H.Zhu,X.Zhou,R.Lo,A.Sridhar,X.Cheng,T.Ou,Y.Bisk,D.Fried,etal. WebArena: A realistic web environment for building autonomous agents. InInternational Conference on Learning Representations, pages 15585–15606, 2024 on Pith
- [32] M. Zhuge, C. Zhao, D. Ashley, W. Wang, D. Khizbullin, Y. Xiong, Z. Liu, E. Chang, R. Krishnamoorthi, Y. Tian, et al. Agent-as-a-judge: Evaluate agents with agents.arXiv preprint arXiv:2410.10934, 2024. 16 EverydayBench: Evaluating Satisfactory Completion of Live User-Facing Di... on PithPith's reviewarXiv
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment