Pith.

Artificial Intelligence · arXiv:2611.25146 v1 · submitted 18 Nov 2026 · 32 references

EverydayBench: Evaluating Satisfactory Completion of Live User-Facing Digital Work

Fangrui Huang, Bingyang Wang, Chuhan Li, Yijiang Li, Arpandeep Khatua, Austin Shen, Riki R Z Choi, Cuiyuanxiu Chen

show 9 more authors

Jin Huang, Zachary Robertson, Kate Lin, Akshay Paruchuri, Sijun Tan, Xin Liu, Zhengmin Ni, Sanmi Koyejo, Ehsan Adeli

The paper says

“Everyday digital tasks, from selecting products and planning trips to conducting research and creating artifacts, require AI agents to deliver useful outcomes rather than merely retrieve web information.”

more of the abstract

Yet evaluating these outcomes on the live web is challenging, as a single request may admit multiple valid results, and changing web information can alter which results remain valid over time. Although existing benchmarks address aspects of this setting, they do not fully integrate predefined success criteria, current evidence verification, and deliverable-quality assessment for agent-selected outcomes. To address this gap, we introduce EverydayBench, a benchmark of 77 human-authored tasks spanning eight domains of everyday life and work. Each task specifies success criteria before execution without prescribing an answer or workflow, allowing agents to select their own outcomes. To evaluate submissions under fixed success criteria, EverydayBench combines (1) deterministic checks of explicit constraints, (2) verification of claims against agent-cited webpages retrieved at evaluation time, and (3) rubric-based assessment of final deliverable quality. We further validate the protocol through human judgments, repeated scoring under fixed evidence, and controlled grounding failures. Across 23 reported agent-model configurations, Claude Opus 5.5 and Claude Fable 5.1 tie for the highest strict pass rate at 54.55%; a controlled GPT-6 Sol effort sweep ranges from 29.87% at low effort to 45.45% at max effort. Reliability varies substantially by task type: GPT-5.6 Sol leads lookup, Fable 5.1 leads planning, Opus 5.5 and Fable 5.1 tie on comparison, and GPT-6 Astra and Sonnet 5.5 tie on customized deliverables. Together, these findings highlight the difficulty of delivering complete, well-supported, and usable outcomes for everyday user requests.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 20 Nov 2026 · read the full text (120,117 characters) · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. EverydayBench is a well-validated benchmark for evaluating whether AI agents deliver satisfactory outcomes on live-web everyday tasks, with a careful evaluation methodology that combines deterministic checks, source-grounded verification, and rubric-based scoring. This line is wrong

8 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

The paper, first character to last

The reviewer was given the whole text, 120,117 characters. Not checked: figures, code, what the 32 references say.

8 minor objections

The paper says, at Abstract

Pith says: minor objection 1 of 8

The abstract mentions '23 reported agent-model configurations' but the main results table (Table 1) shows 23 configurations including the effort sweep. The count is correct but the phrasing could be clearer about which configurations are in the primary leaderboard vs. the effort sweep.

The paper says, at §3.2, Evaluation Setup

Pith says: minor objection 2 of 8

The paper states 'The judging backbone is GPT-5.4 at temperature zero' but does not discuss potential biases of using an LLM as a judge, even though it validates against human judgments. A brief discussion of the limitations of LLM-as-a-judge would strengthen the paper.

show 6 more

The paper says, at §4.2, Table 1

Pith says: minor objection 3 of 8

The table caption says 'Bold marks the best pass rate in each column' but the bold entries in the 'Plan' column show 66.67 for Fable 5.1 and 58.33 for several others, which is correct. However, in the 'Build' column, GPT-6 Astra and Sonnet 5.5 both show 76.92 but neither is bolded. The caption says bold marks the best, so both should be bolded.

The paper says, at §4.3, Failure Mode Analysis

Pith says: minor objection 4 of 8

The failure mode analysis is thorough but the description of the failure-mode judge's agreement with human labels (92 of 100) could be reported as a percentage (92%) for consistency with other agreement metrics in the paper.

The paper says, at §5, Appendix A.1

Pith says: minor objection 5 of 8

The pass-threshold sensitivity analysis is valuable but the table would benefit from a note explaining why the 19 configurations in the primary leaderboard exclude the 4 effort-sweep configurations. This is clear from context but could be stated explicitly.

The paper says, at §5, Appendix A.2

Pith says: minor objection 6 of 8

The computer-use vs. retrieval-only comparison on 25 tasks is interesting but the sample size is small. The paper should note that the 25 tasks were sampled and whether they are representative of the full 77-task set.

The paper says, at §5, Appendix B

Pith says: minor objection 7 of 8

The human validation section reports Cohen's kappa values but does not specify whether the kappa values are for exact agreement or weighted agreement. The main text says 'linearly weighted Cohen's κ' but the appendix should also state this clearly.

The paper says, at §5, Appendix C

Pith says: minor objection 8 of 8

The failure mode analysis appendix is very detailed and well-structured. However, the prompts for the failure-mode judge (Prompts 1-3b) are very long and complex. A brief summary of how the prompts were developed and validated would help readers assess the reliability of the failure mode classification.

What would settle it

No significant objection identified

from the skeptic's reading

How the review went

5 passes by deepseek-v4-flash · 20 Nov 2026

  1. Reader holds up confidence high
  2. Skeptic agrees 5 s · 35,151 tokens read, 76 written
  3. Referee 8 minor 13 s · 35,510 tokens read, 1,052 written
  4. Check against the paper
  5. Editor wrote the verdict above 8 s · 35,714 tokens read, 428 written

Not run on this paper: the simulated authors' defence.

Pith checked this review's objections against the text of the paper it read.

The review in full

The review in brief, machine-written

EverydayBench addresses a real gap: most agent benchmarks evaluate whether an agent can retrieve information or perform actions, but not whether the final deliverable is actually useful for the person who asked. The paper's central design choice—fix the standard of completion, not the solution—is sensible, and the three-part evaluation (direct checks, source grounding, rubric scoring) is a genuine methodological contribution.

The validation is thorough. Human agreement studies for both grounding (κ=0.695) and rubric judging (κ=0.644) are reported, temporal stability is examined, and the seeded-fault detection rate of 90.7% gives confidence that the evaluator catches real failures. The failure-mode analysis is particularly valuable: it shows concretely where agents lose points, and the finding that even top configurations lose most points because "a claim says more than the page it cites" is both specific and actionable.

The main limitation is inherent to the approach: 77 tasks, however carefully curated, cannot cover the full space of everyday digital work. The authors acknowledge this. A second concern is that the LLM-as-a-judge component, while validated against humans, still relies on GPT-5.4 as the backbone; the paper's own ablation shows that the full evaluator is more stable than simpler baselines, but the judge's reliability on tasks far from its training distribution is untested. These are minor caveats for a benchmark paper.

This paper is for anyone building or evaluating web agents who wants to move beyond retrieval accuracy toward outcome quality. It deserves a serious referee and would benefit from community adoption and extension.

Recommendation: accept with minor revisions. The methodology is sound, the validation is adequate, and the benchmark fills a clear gap.

The strongest objection

No significant objection identified

No significant objection identified

The referee's report

The benchmark addresses a real gap in evaluating AI agents for everyday digital work on the live web, where multiple valid outcomes exist and web content changes over time. The evaluation methodology combining deterministic checks, source verification, and human-validated rubrics is well-designed and validated. The failure mode analysis provides actionable insights for improving agent systems. The benchmark is likely to be useful for the community.

Pith's reading of the paper (a machine's summary, not the authors')

The paper argues that evaluating AI agents on everyday digital work should be judged by whether the delivered outcome is satisfactory — it meets the stated requirements, is usable for its intended purpose, and is supported by current web evidence — rather than by whether it matches a reference answer. To make that standard measurable, it introduces EverydayBench, a benchmark of 77 human-authored tasks across eight domains, where success criteria are fixed before execution but agents are free to choose their own sources, workflows, and concrete results. Each submission is scored by combining deterministic checks of explicit constraints, verification of claims against the live pages the agent cites, and rubric-based scoring of the final deliverable's quality, an assessment validated against independent human judgments. Across 23 agent–model configurations, the best strict pass rates tie at 54.55%, and only 32.30% of all runs meet the benchmark's completion standard, evidence that producing plausible-looking information and delivering complete, usable, supported work are different things.

What carries the argument. The load-bearing mechanism is the three-route criterion verification over an outcome schema. Each of a task's success criteria (7.6 per request on average) is assigned to one of three routes: deterministic direct tests for properties checkable from the submission alone (arithmetic, cross-field consistency, artifact structure); source-grounded checks that determine whether a submitted factual claim is supported by the exact agent-cited page retrieved at evaluation time; and rubric-based component scoring in which each predefined outcome component is judged on a 1–5 scale against human-curated anchors for scores 1, 3, and 5, with a format-violating output scoring 0 on every component. Component scores are weight-averaged and scaled to $[0,1]$, and a task passes only when its overall score is at least 85% and no component scores below 3.

Core claim. EverydayBench's central claim is that satisfactory task completion can be operationalized as an outcome-level standard: fix the completion criteria before execution while leaving the solution open, then assess the agent-selected outcome through three complementary routes — deterministic checks of explicit constraints, verification of factual claims against the exact webpages the agent cites retrieved at evaluation time (with an inaccessible page yielding an unverified finding rather than a factual rejection), and rubric-based component scoring with human-curated 1/3/5 anchors. Under this protocol, judged by GPT-5.4 at temperature zero, the strongest agents (Claude Code with Opus 5.5 and with Fable 5.1) tie for first at a 54.55% strict pass rate, and only 32.30% of 1,771 agent–task runs pass. The failure analysis shows that the most common way even frontier agents lose points is a claim that says more than the page it cites, while lower-cost agents lose points mainly through output files that are missing or violate the required format and through user needs the model never investigates.

The premise it rests on. The load-bearing premise is that the 77 human-authored tasks are representative enough of the broad space of everyday digital work, and that the evaluation protocol — deterministic checks, source-grounding against live cited pages, and rubric scoring validated against human judges — measures satisfactory completion validly and reliably for those tasks.

If the paper is right (Pith's inferences, not the authors')

If the benchmark is valid, it gives the field a yardstick for everyday agent work that survives web change: because claims are checked against live pages at evaluation time, a score reflects whether the outcome is still valid when judged, not only when produced.

The reported validation numbers — mean human–evaluator agreement of $\kappa=0.695$ on grounding and $\kappa=0.644$ on rubric scoring, close to or above human–human agreement — support treating the judge's scores as reproducible for benchmarking comparisons.

The result that no configuration leads across all five task types (lookup, comparison, planning, analysis, build) implies systems should be selected by the mix of work an application emphasizes, not by aggregate ranking alone.

The system-wide 32.30% pass rate sets a concrete target: current agents complete everyday delegated work at most about half the time even at the frontier, and additional reasoning effort raises the upper envelope of reliability rather than monotonically guaranteeing more completed tasks.

The finding that harness choice with a fixed model changes pass rate by 8–20 points implies agentic infrastructure, not only the underlying model, is a lever on satisfactory completion.

A natural extension the authors do not pursue: whether the same three-route protocol transfers to safety- or compliance-critical delegation, where an unacted-upon implicit need (like the short-term-rental ordinance example) carries higher stakes than in shopping or trip planning.

The paper's own failure taxonomy suggests a testable hypothesis beyond its scope: the dominant frontier failure, 'claim says more than the cited page,' is concentrated in how claims are phrased in the final deliverable rather than in the research itself, which would imply that grounding-aware generation or post-hoc claim-verification of deliverables could recover many lost points.

Because the assessor is a single frozen judge (GPT-5.4), the reported 54.55% is one judge's reading; a robustness check not done in the paper would be scoring the same 1,771 runs with a judge of a different model family and with an independent human panel to see whether the leaderboard ordering and the 32.30% aggregate survive.

Check this review

What was read

The passes were given all 120,117 characters of the paper's LaTeX source.

Fingerprint of that text (SHA-256): 6fb9046be597b35164159190a8efa8179294251e0665bb6e57c04d39b0364fb1

Pith does not keep that text. Rebuilt from the paper's source or PDF by the same code, it has this fingerprint if it is the same text. From a copy of Pith's code:

PYTHONPATH=. python scripts/review_input_check.py 2611.25146

The passes, in the order they ran

  1. Reader deepseek-v4-flash · medium effort · $0.0056 · instructions of the Reader, fingerprint f9ead7300382…
  2. Skeptic deepseek-v4-flash · medium effort · $0.0031 · instructions of the Skeptic, fingerprint 8f88c0f61488…
  3. Referee deepseek-v4-flash · medium effort · $0.0034 · instructions of the Referee, fingerprint a852d99aea0d…
  4. Editor deepseek-v4-flash · medium effort · $0.0033 · instructions of the Editor, fingerprint e7211f6a3d27…
  5. Summary for the page deepseek-v4-flash · medium effort · temperature 0.2 · $0.0066 · instructions of the Summary for the page, fingerprint e4f56bf568e7…

The passes cost $0.022 in model calls together.

Version stamped on this review: pith-pipeline@v1.4.1-daily-deepseek-or-medium+deepseek/deepseek-v4-flash+p.448e4af8143a

Each link opens a pass's instructions as plain text, the exact bytes its fingerprint was taken over. Every stored set of instructions.

curl -s https://pith.science/methodology/prompts/f9ead730038216984cfb56d7cb34332bdf691828296a1de4245f3d6667d4bf56 | sha256sum

prints the fingerprint of the reader's instructions. The same works for every pass.

The receipt

The review's fingerprint was taken on 20 Nov 2026. The review you are reading still matches it.

Fingerprint of the review (SHA-256): ecafe8e23ba4d51b20ca87cf1ae884d2b2456fdaa691d12af0c982e68fcd9de4

curl -s 'https://pith.science/api/v1/verdicts/2611.25146?form=canonical' | sha256sum

prints the fingerprint of the review as it stands now. The rule, in four steps.

A matching fingerprint shows the review has not changed since that date. It does not show that the review is right.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, from the paper's LaTeX source.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Everyday digital tasks, from selecting products and planning trips to conducting research and creating artifacts, require AI agents to deliver useful outcomes rather than merely retrieve web information. Yet evaluating these outcomes on the live web is challenging, as a single request may admit multiple valid results, and changing web information can alter which results remain valid over time. Although existing benchmarks address aspects of this setting, they do not fully integrate predefined success criteria, current evidence verification, and deliverable-quality assessment for agent-selected outcomes. To address this gap, we introduce EverydayBench, a benchmark of 77 human-authored tasks spanning eight domains of everyday life and work. Each task specifies success criteria before execution without prescribing an answer or workflow, allowing agents to select their own outcomes. To evaluate submissions under fixed success criteria, EverydayBench combines (1) deterministic checks of explicit constraints, (2) verification of claims against agent-cited webpages retrieved at evaluation time, and (3) rubric-based assessment of final deliverable quality. We further validate the protocol through human judgments, repeated scoring under fixed evidence, and controlled grounding failures. Across 23 reported agent-model configurations, Claude Opus 5.5 and Claude Fable 5.1 tie for the highest strict pass rate at 54.55%; a controlled GPT-6 Sol effort sweep ranges from 29.87% at low effort to 45.45% at max effort. Reliability varies substantially by task type: GPT-5.6 Sol leads lookup, Fable 5.1 leads planning, Opus 5.5 and Fable 5.1 tie on comparison, and GPT-6 Astra and Sonnet 5.5 tie on customized deliverables. Together, these findings highlight the difficulty of delivering complete, well-supported, and usable outcomes for everyday user requests.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-20

The authors' main claim, as the classifier picked it out 1.00

“Together, these findings highlight the difficulty of delivering complete, well-supported, and usable outcomes for everyday user requests.”

How the main claim is stated: asserted 0.97

What the abstract itself states, as probabilities: claims first 0.13 · states limitation 0.45 · human participants 0.72 · controlled comparison 0.31 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.71 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Everyday digital tasks, from selecting products and planning trips to conducting research and creating artifacts, require AI agents to deliver useful outcomes rather than merely retrieve web information. background 1.00 0.06 0.05 asserted 1.00
Yet evaluating these outcomes on the live web is challenging, as a single request may admit multiple valid results, and changing web information can alter which results remain valid over time. background 0.92 0.08 0.03 asserted 0.98
Although existing benchmarks address aspects of this setting, they do not fully integrate predefined success criteria, current evidence verification, and deliverable-quality assessment for agent-selected outcomes. gap 1.00 0.07 0.05 asserted 1.00
To address this gap, we introduce EverydayBench, a benchmark of 77 human-authored tasks spanning eight domains of everyday life and work. contribution 0.98 0.92 0.04 asserted 1.00
Each task specifies success criteria before execution without prescribing an answer or workflow, allowing agents to select their own outcomes. method 0.79 0.82 0.03 asserted 1.00
To evaluate submissions under fixed success criteria, EverydayBench combines (1) deterministic checks of explicit constraints, (2) verification of claims against agent-cited webpages retrieved at evaluation time, and (3) rubric-based assessment of final deliverable quality. method 0.97 0.90 0.05 asserted 1.00
We further validate the protocol through human judgments, repeated scoring under fixed evidence, and controlled grounding failures. method 0.93 0.95 0.03 asserted 1.00
Across 23 reported agent-model configurations, Claude Opus 5.5 and Claude Fable 5.1 tie for the highest strict pass rate at 54.55%; a controlled GPT-6 Sol effort sweep ranges from 29.87% at low effort to 45.45% at max effort. result 1.00 0.95 0.09 asserted 1.00
Reliability varies substantially by task type: GPT-5.6 Sol leads lookup, Fable 5.1 leads planning, Opus 5.5 and Fable 5.1 tie on comparison, and GPT-6 Astra and Sonnet 5.5 tie on customized deliverables. result 1.00 0.94 0.07 asserted 1.00
Together, these findings highlight the difficulty of delivering complete, well-supported, and usable outcomes for everyday user requests. interpretation 1.00 0.80 0.82 asserted 0.97

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-20: 1 request, 7,829 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (9)
Figure 1 of the paper
Figure 1. Overview of the EverydayBench framework. A user request is paired with fixed success criteria; agents may submit different outcomes, which are evaluated through deterministic direct tests, source-grounded checks, and rubric-based component scoring. satisfy the stated requirements, provide a deliverable usable for its intended purpose, and adequately support material claims requiring external evidence. This notion concerns observable properties of the work product, rather than a user’s subjective satisfaction or unexpressed preferences. Therefore, it provides an operational target…
Figure 2 of the paper
Figure 2. Standard fixed-answer curation and our success-criteria curation. Human curators translate an open-ended user request into criterion-level direct tests, grounding requirements, and weighted 1/3/5 rubric anchors, allowing multiple valid outcomes while making specific failures observable. 4. Evaluation Setup and Validation 4.1. Evaluation Setup We evaluate 23 agent–model configurations on 77 tasks, yielding 1,771 agent runs and corresponding judgments. With official Harbor 0.24.0, each agent is run in an isolated container using a native CLI start and multi-turn resume, with 3,000…
Figure 3 of the paper
Figure 3. Panel (a) compares multiple agent runs with near-synchronized judging at different points in time to validate benchmark stability; panel (b) holds agent run fixed to show time-sensitivity to web contents under near-synchronized, 20-day￾delayed, and 30-day-delayed judging. Scoring stability despite time-sensitive judge. EverydayBench judges each submitted claim against the live version of the page the agent cites, so a score reflects whether an outcome is still valid when it is judged, not only when it was produced. As web content changes, for example when a price, a review count, or…
Figure 4 of the paper
Figure 4. Seeded-fault detection by judges. Our full evaluator has a significantly more stable mean within-trace SD of 1.41%, compared with 2.64% for rubric￾only scoring and 4.51% for one-prompt scoring (
Figure 5 of the paper
Figure 5. Deductions per 100 components by failure type for the frontier and low-cost groups over the 77 tasks. A deduction is a component that scores below 5. Numbers in parentheses are the category totals for both groups. Intent recognition. Intent recognition accounts for 16.5 deductions per 100 components in the frontier group and 22.3 in the low-cost group. In these deduc￾tions, the output meets the explicit request but misses an implicit need of the user. We further classify these deductions into two failure types by searching the agent’s trace and output files: the model does not look
Figure 6 of the paper
Figure 6. Strict pass rate against runtime and inferred model cost. The connected blue path is the controlled Codex/GPT-6 Sol effort sweep from low to max. Cost combines uncached input, cache reads and writes, and output tokens at public list prices; the logarithmic axis exposes the full price range. Subscription-authenticated runs are normalized to the corresponding API-equivalent rate rather than treated as cash expenditure. Spending more time does not by itself produce more completed tasks. Across the 23 configurations, mean runtime is only moderately associated with pass rate (𝜌 = 0.45, 𝑝…
Figure 7 of the paper
Figure 7. Three execution trajectories from the computer-use runs. The final row is sampled from the InsureMyTrip browser recording and shows the agent progressively entering quote-search information. 19
Figure 8 of the paper
Figure 8. Human validation interface for grounding agents. 21
Figure 9 of the paper
Figure 9. Human validation interface for rubric judge. 22

Figures are the authors', reproduced from the paper.

References (32)
  1. [1] Alibaba Cloud. Model studio model inference pricing.https://www.alibabacloud.com/help/ en/model-studio/model-pricing, 2026. Accessed November 5, 2026 on Pith
  2. [2] Anthropic. Claude model pricing.https://www.anthropic.com/pricing#api, 2026. Accessed November 5, 2026 on Pith
  3. [3] R. K. Arora, J. Wei, R. S. Hicks, P. Bowman, J. Quiñonero-Candela, F. Tsimpourlas, M. Sharman, M. Shah, A. Vallone, A. Beutel, et al. HealthBench: Evaluating large language models towards improved human health.arXiv preprint arXiv:2505.08775, 2025 on PithPith's reviewarXiv
  4. [4] C. Bandi, R.-G. Dumitru, B. Hertzberg, D. Agarwal, G. Boo, T. Polakam, S. Hassaan, J. Da, H. Kim, V. Gupta, M. Sharma, A. Park, M. Dimakis, E. G. H. Montoya, D. Rambado, I. Salazar, R. Cruz, M.Rezaei, C.Rane, B.Levin, D.Y.Zhang, B.Kenstler, andB.Liu. Mcp-atlas: Alarge-scaleben... on Pith
  5. [5] J. Cohen. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychologicalbulletin, 70(4):213, 1968 on Pith
  6. [6] X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su. Mind2Web: Towards a generalistagentfortheweb. AdvancesinNeuralInformationProcessingSystems, 36:28091–28114, 2023 on Pith
  7. [7] M. Du, B. Xu, C. Zhu, L. Zhang, X. Wang, and Z. Mao. DeepResearch Bench: A comprehensive benchmark for deep research agents. InInternational Conference on Learning Representations, pages 42414–42448, 2026 on Pith
  8. [8] B. Gou, Z. Huang, Y. Ning, Y. Gu, M. Lin, W. Qi, A. Kopanev, B. Yu, B. Jimenez Gutierrez, Y. Shu, et al. Mind2Web 2: Evaluating agentic search with agent-as-a-judge.Advances in Neural Information Processing Systems, 38, 2026 on Pith
  9. [9] H. He, W. Yao, K. Ma, W. Yu, Y. Dai, H. Zhang, Z. Lan, and D. Yu. WebVoyager: Building an end-to- end web agent with large multimodal models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume1: Long Papers), pages 6864–6890, 2024 on Pith
  10. [10] F. Huang, S. Chbeir, A. Khatua, S. Wang, S. Tan, K. Ye, L. Bailey, M. Daniel, R. Louie, S. Koyejo, and E. Adeli. TherapyGym: Evaluating and aligning clinical fidelity and safety in therapy chatbots. In Advances in Neural InformationProcessing Systems (Evaluationsand Datasets T... on Pith
  11. [11] L. K. Jang, J. Y. Koh, D. Fried, and R. Salakhutdinov. Odysseys: Benchmarking web agents on realistic long horizon tasks.arXiv preprint arXiv:2604.24964, 2026 on PithPith's reviewarXiv
  12. [12] Z. Jiang, F. Huang, H. Xing, X. Wu, Y. Gao, R. Cao, M. Wang, S. Liu, and Y. Li. Demystifying agent skills: Why they work—until they don’t.arXiv preprint arXiv:2608.14036, 2026 on PithPith's reviewarXiv
  13. [13] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, pages 54107–54157, 2024 on Pith
  14. [14] J. Li, W. Zhao, J. Zhao, W. Zeng, H. Wu, X. Wang, R. Ge, Y. Cao, Y. Huang, W. Liu, et al. The Tool Decathlon: Benchmarking language agents for diverse, realistic, and long-horizon task execution. In International Conference on Learning Representations, pages 53359–53394, 2026 on Pith
  15. [15] X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, et al. AgentBench: 14 EverydayBench: Evaluating Satisfactory Completion of Live User-Facing Digital Work Evaluating LLMs as agents. InInternationalConferenceonLearning Representations, pages 52989... on Pith
  16. [16] Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu. G-Eval: NLG evaluation using GPT-4 with better human alignment. InProceedings of the 2023 conference on empirical methods in natural language processing, pages 2511–2522, 2023 on Pith
  17. [17] G. Mialon, C. Fourrier, T. Wolf, Y. LeCun, and T. Scialom. GAIA: a benchmark for general AI assistants. InInternational Conference on Learning Representations, pages 9025–9049, 2024 on Pith
  18. [18] S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 conference on empirical methods in natural language pro... on Pith
  19. [19] OpenAI. Api pricing. https://developers.openai.com/api/docs/pricing, 2026. Accessed November 5, 2026 on Pith
  20. [20] T. Patwardhan, R. Dias, E. Proehl, G. Kim, M. Wang, O. Watkins, S. Fishman, M. Aljubeh, P. Thacker, L. Fauconnet, et al. GDPval: Evaluating AI model performance on real-world economically valuable tasks. InInternational Conference on Learning Representations, pages 24005–24040, 2026 on Pith
show 12 more
  1. [21] M. Sharma, C. B. C. Zhang, C. Bandi, C. Wang, A. Aich, H. Nghiem, T. Rabbani, Y. Htet, B. Jang, S. Basu, et al. ResearchRubrics: A benchmark of prompts and rubrics for evaluating deep research agents. InInternational Conference on Learning Representations, pages 90447–90472, 2026 on Pith
  2. [22] G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson, et al. PaperBench: evaluating AI’s ability to replicate AI research.arXiv preprint arXiv:2504.01848, 2025 on PithPith's reviewarXiv
  3. [23] C. Team, S. Hao, Z. Zhang, Z. Liang, T. Liu, Y. Zha, Q. Gao, J. Chen, Z. Wang, Z. Cheng, et al. CocoaBench: Evaluating unified digital agents in the wild.arXiv preprint arXiv:2604.11201, 2026 on PithPith's reviewarXiv
  4. [24] J. Wang, Y. Ming, R. Dulepet, Q. Chen, A. Xu, Z. Ke, F. Sala, A. Albarghouthi, C. Xiong, and S. Joty. LiveResearchBench: A live benchmark for user-centric deep research in the wild. InInternational Conference on Learning Representations, pages 95096–95143, 2026 on Pith
  5. [25] J. Wei, Z. Sun, S. Papay, S. McKinney, J. Han, I. Fulford, H. W. Chung, A. T. Passos, W. Fedus, and A. Glaese. BrowseComp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516, 2025 on PithPith's reviewarXiv
  6. [26] T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, et al. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural InformationProcessing Systems, 37:52040–52094, 2024 on Pith
  7. [27] T. Xue, W. Qi, T. Shi, C. H. Song, B. Gou, D. Song, H. Sun, and Y. Su. An illusion of progress? assessing the current state of web agents.arXiv preprint arXiv:2504.01382, 2025 on PitharXiv
  8. [28] S.Yao,N.Shinn,P.Razavi,andK.Narasimhan. 𝜏-bench: Abenchmarkfortool-agent-userinteraction in real-world domains.arXiv preprint arXiv:2406.12045, 2024 on PithPith's reviewarXiv
  9. [29] O. Yoran, S. J. Amouyal, C. Malaviya, B. Bogin, O. Press, and J. Berant. AssistantBench: Can web agents solve realistic and time-consuming tasks? InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8938–8968, 2024. 15 EverydayBench: ... on Pith
  10. [30] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.Advances in neural information processing systems, 36:46595–46623, 2023 on Pith
  11. [31] S.Zhou,F.F.Xu,H.Zhu,X.Zhou,R.Lo,A.Sridhar,X.Cheng,T.Ou,Y.Bisk,D.Fried,etal. WebArena: A realistic web environment for building autonomous agents. InInternational Conference on Learning Representations, pages 15585–15606, 2024 on Pith
  12. [32] M. Zhuge, C. Zhao, D. Ashley, W. Wang, D. Khizbullin, Y. Xiong, Z. Liu, E. Chang, R. Krishnamoorthi, Y. Tian, et al. Agent-as-a-judge: Evaluate agents with agents.arXiv preprint arXiv:2410.10934, 2024. 16 EverydayBench: Evaluating Satisfactory Completion of Live User-Facing Di... on PithPith's reviewarXiv

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment