Pith. sign in

REVIEW 5 major objections 5 minor 25 references

Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read WorkBuddy Bench claims that reverse-engineering real commits into colloquial prompts makes a fully open benchmark resistant to web-search contamination.

desk verdict A genuinely useful, well-documented benchmark construction, but the two load-bearing claims — unsearchable prompts and an auditable open release — are asserted rather than demonstrated in the manuscript. read the letter →

arxiv 2607.20911 v1 pith:SZBYRJKI submitted 2026-07-23 cs.CL cs.SE

Tencent WorkBuddy Bench Team: Siqi Cai , Shaopeng Chen , Xiang Fei , Yong Mao , Zihan Xu , Zhiheng Lyu , Zhijian Shao , Yuchen Shi
show 29 more authors
Shuwen Zhang Chaofan Qiu Linjie Che Xiaoxi Zhao Feng Wu Kai Zhang Chaofan Zhu Yubin Qi Xiaoyun Liang Peijie Dong Yunhao Zhang Yuanjie Zhu Ling Jiang Xianjun Zhang Zhehang Chu Anyuan Sang Zhen Feng Sen Nie Shi Wu Yuanzhen Xu Xin Li Ning Yang Zhiqiang Dong Hande Dong Qiang Lin Yi Liu Yunsheng Wu Ke Li Xing Sun
This is my paper · ORCID
classification cs.CLcs.SE
keywords multi-domaincoding-agentbenchmarkcontamination-resistantconstructionrole-playedtaskrewritingopenauditabilityrepository-levelsoftwareengineeringwebfront-endevaluationofficeworkflowsecurityagent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

WorkBuddy Bench is an open, 260-task evaluation suite for coding agents that tries to solve the contamination problem at the point of task writing rather than by hiding the dataset. Each task is reverse-engineered from a real commit, pull request, CVE, or business scenario and rewritten as a short, colloquial, deliberately underspecified request, so a web search for the underlying issue or PR does not recover the prompt. The suite spans four work domains, Code, Web, Office, and Security, under one task-directory format and one sandboxed execution harness, with each domain scored by its own instrument and no suite-wide average. The full release includes task directories, environment images, grading tests, and reference solutions, so any third party can re-run a task and audit the result. If the construction claim holds, the benchmark offers a reproducible, auditable alternative to both memorization-prone public suites and closed vendor benchmarks.

What carries the argument

The mechanism that carries the argument is a construction protocol rather than a single theorem: each task is anchored to a real upstream artifact, such as a historical commit, pull request, or CVE, or to an abstracted business scenario, and then rewritten as a short, colloquial, deliberately underspecified request voiced through a requester persona, with the root cause and reference diff withheld from the agent. This is what makes the prompt unavailable to web search by construction. The second carrier is a Harbor-style task-directory format in which the agent-visible Docker workspace contains only the repository or business files, while grading assets live in a separate tests/ directory and are introduced only after the episode ends. Code tasks additionally pass an oracle-gated admission check, with baseline reward $\le 0.3$ and oracle reward $= 1.0$, before release.

What would settle it

For each task in the released 260-task suite, take the instruction text and query a general web search engine, as well as a large code corpus, for the originating repository, commit hash, pull-request number, or CVE description; then count how many tasks return their source thread among the top results. If a substantial fraction do, the claim that prompts are unrecoverable by web-searching fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that a benchmark can be both fully open and resistant to the contamination path that matters most, web-searchable prompts, if the prompts are constructed rather than collected. Every task is anchored to a concrete origin, a historical commit or pull request in an open-source repository, a historical CVE, or an abstracted business scenario, and rewritten as a short, colloquial, role-played request that withholds the root cause, the reference diff, and any solution-hinting framing. Grading assets are introduced only after the agent has finished acting, and the released package contains everything needed to reproduce and audit a run. The paper also claims that the distribution of tasks, in categories, modes, roles, and difficulty, is matched to internal usage taxonomies of real requests, and that this is done without using or releasing any raw user data. Because the four subsets use different scoring instruments, the suite reports per-subset scores and deliberately no suite-wide average.

Load-bearing premise

The load-bearing premise is that colloquially rewriting a real commit or CVE makes each prompt unrecoverable by web-searching the source thread, a property the paper asserts by construction without a retrieval or corpus-overlap audit.

Editorial extensions

If this is right

  • Any third party can re-run every task and inspect the grading tests and reference solutions, so reported scores are auditable rather than trusted from a closed vendor.
  • Because prompts are written from reverse-engineered artifacts rather than copied from issue text, memorizing the underlying commit or pull-request thread does not by itself hand the agent the answer at release time.
  • Rankings depend on the harness: the paper's leaderboard shows model order shifting between CodeBuddy Code and Claude Code, most strongly on Security, so a single overall ranking would not be meaningful.
  • The four subsets are separate instruments with different scoring rules, so scores should be compared only within a subset; the suite intentionally reports no suite-wide average.
  • Code's difficulty concentrates in bug-fix and api-contract tasks, where colloquial requests must be grounded in the right lines of a repository, a finding consistent with the paper's view that navigation and understanding, not synthesis, limit current models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the reverse-engineer-and-rewrite recipe is domain-neutral, so the same template could be carried to notebook data science, DevOps runbooks, or design systems; the durable contribution may be the construction format rather than the four current domains.
  • My inference: the central contamination claim would become measurable if the project ran a retrieval audit at each release, searching each prompt against its source commit, PR, or CVE corpus, and published the match rate.
  • My inference: because the internal usage taxonomies behind the task mix are not disclosed, representativeness is currently an assumption; publishing those taxonomies in aggregate form would turn it into a testable claim.
  • My inference: the observed harness-dependent reordering suggests benchmark consumers should treat a leaderboard as bound to the harness, not as a model-quality fact.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper introduces Tencent WorkBuddy Bench, a 260-task benchmark (Code 80, Web 70, Office 50, Security 60) for coding agents. The main methodological proposal is that every task is reverse-engineered from a real upstream commit, CVE, or business scenario and rewritten as a short, colloquial, role-played instruction, so that the prompt text cannot be found by web-searching the underlying source thread; the suite is then released openly with task directories, images, harness, tests, and reference solutions, and contamination resistance rests on construction plus versioning. Each track has its own verifier (hidden tests, rubric judges, rule/Judge blend, deterministic scorer), scores are not aggregated across tracks by design, and results are reported for seven models under two harnesses. The paper also reports token/turn efficiency and per-category analyses.

Significance. The benchmark fills a real gap if its two load-bearing claims hold: construction-level resistance to web-searchable prompts and a fully open release enabling third-party audit. The paper is unusually candid, disclosing judge-bias risk, post-release contamination exposure, the modified-instruction cell, and single-language bias; the Security scorer is deterministic with anti-cheat layers, and the Office/Web scoring equations are explicit. However, the contamination-resistance claim is a design assertion without a retrieval audit, the open-release claim has no artifact identifier, and Figure 1 contradicts the stated no-overall-score principle. The contribution is therefore significant but conditional on verification and repair of these points.

major comments (5)
  1. [§1; Figure 1] Section 1 and Section 2 state repeatedly that scores are not comparable across subsets and that the suite reports no suite-wide average, yet Figure 1 displays an 'Overall Score' column with ranks (e.g., Claude Opus 4.8 at 75.0%). The listed values do not match any disclosed aggregation of Table 6 (the unweighted mean of that model's four Table 6 scores is 72.3), so the figure either contradicts the stated design principle or rests on an undocumented weighting. Please remove the column or provide the aggregation formula and justify its use.
  2. [§2, 'Contamination-resistant task construction'] Section 2's 'Contamination-resistant task construction' paragraph asserts that no task instruction is recoverable by web-searching the underlying commit, issue, or PR thread, but this is argued by construction rather than demonstrated. The manuscript reports no retrieval or corpus-overlap audit, no search-query construction, and no n-gram or near-duplicate statistics, even though rewritten prompts can retain identifiers, code symbols, or distinctive phrasing from their source artifacts. Because this claim is load-bearing for the benchmark's central value, please add a reproducible retrieval audit over the released instruction texts against the source threads and a web corpus, and report overlap rates and retrieval recall.
  3. [§2.1; Table 2] Section 2.1 and Table 2 claim that the suite is released fully open — task directories, environment images, evaluation harness, grading tests, and reference solutions — and that any third party can re-run each task. However, the manuscript contains no repository URL, dataset DOI, commit hash, or release identifier for WorkBuddy Bench itself; the only artifact links given are for the Harbor framework (reference [6]). Without a persistent identifier and version, the auditability and reproducibility claims have no concrete object. Please provide the release URL/DOI, commit hash, and checksums for the task directories and images.
  4. [§2; §3.3] Section 2 says each subset's category mix, modes, roles, and difficulty are matched to the distribution of real requests using internal usage taxonomies, but neither the taxonomies nor the aggregation method is disclosed, and Section 3.3 explicitly states that the Office scenario groups 'describe benchmark coverage rather than estimate production request traffic.' This is a direct tension in the representativeness claim. Please either release the taxonomy and sampling/validation procedure or soften the distribution-matched framing to a coverage-design claim.
  5. [§5; Table 6] Table 6 and Section 5 report three-run means without per-run scores, variances, or confidence intervals, while making strong rank and harness-sensitivity claims (e.g., the Security mean absolute shift of 8.6 points and the reordering of GPT-5.5 from sixth to second). Given known run-to-run variability in agent evaluations, these claims need uncertainty quantification; at minimum, provide per-run scores or bootstrap confidence intervals. Please also state explicitly how Security refusals (e.g., Claude Opus 4.8's 13 under Claude Code) are counted in the reported reward.
minor comments (5)
  1. [§1; §3.1] The Contributions section states the admission gate requires 'oracle reward ≥ 1.0,' while Section 3.1 says 'oracle reward = 1.0'; make the two statements consistent.
  2. [Figure 1; Table 6] Figure 1's leaderboard includes Kimi K2.7 (Security '—'), but Table 6 omits Kimi and includes DeepSeek-V4-Flash; clarify which models are in the official leaderboard and why.
  3. [§5.1] The footnote to the Security turn statistics says they still use an earlier turn-counting convention; since the text compares Security with Code, Web, and Office, either recompute Security under the same convention or explicitly mark all Security efficiency figures as non-comparable.
  4. [§4, Eq. (2)] Equation (2) should state whether the configured penalties p_{t,i} are normalized so that the max(0, 1 − sum) expression has a consistent scale across tasks, and whether fatal failures also incur a configured penalty.
  5. [§2, versioning paragraph] The canary-string mechanism is mentioned only in passing; a short technical description of the format, insertion policy, and detection method would make the versioning claim concrete and auditable.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the scoring chain is deterministic or evidence-grounded, and the contamination-resistance and open-release claims are unverified empirical assertions rather than reductions to inputs.

full rationale

WorkBuddy Bench's scoring chain is self-contained: the Code headline metric is the per-run average of hidden-test pass rates computed through the external Harbor harness (Eq. 1); Office combines deterministic rule checks with a binary, evidence-grounded LLM judge using preconfigured per-task weights (Eqs. 3-6); Security uses a per-task deterministic scoring.py with no LLM judge; Web uses rule, LLM/VLM, and agent-judge rubric items with fixed penalties. No fitted parameter is later reported as a prediction, and the admission gates (baseline <= 0.3, oracle = 1.0) are calibration filters rather than derived outputs. The two load-bearing claims about the benchmark itself are empirical or evidentiary assertions, not circular reductions: Section 2 asserts that colloquial rewriting makes prompts unrecoverable by web search without reporting a retrieval or corpus-overlap audit, and Section 2.1/Table 2 list release components without a URL, commit hash, or DOI. Both are verification gaps, and the paper candly discloses residual post-release contamination exposure and model-judge bias in Section 7. No load-bearing self-citation or imported uniqueness theorem appears; the use of CodeBuddy Code as a harness and the first-party HY serving endpoint are disclosed measurement conditions rather than fitted inputs. The derivation of leaderboard numbers from task verifiers does not reduce to any input by construction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central results rest on several unverified domain assumptions: internal usage taxonomies represent real work, colloquial rewriting prevents search leakage, post-episode isolation holds, and pretraining freshness holds. The main numeric free parameters are scoring weights and admission thresholds. No new physical or mathematical entity is introduced.

free parameters (4)
  • Office rule weight w_i = 0.70-0.95, per task
    Equation (5) blends the deterministic Rule score and the LLM Judge score with a task-preconfigured weight. The value is chosen by hand and directly determines the Office leaderboard; no principled calibration is reported.
  • Web rubric penalties p_{t,i} = 0.1 / 0.2 / 0.3
    Equation (2) subtracts configured penalties for failed non-fatal rubric items. Values are hand-set per task and directly determine Web task rewards.
  • Security scorer weights w1, w2, w3 = not specified in the text
    Section 4 defines the Security reward as w1*artifact + w2*correctness + w3*robustness but does not give the weights, so Security scores cannot be recomputed from the report alone.
  • Admission gate thresholds = baseline <= 0.3, oracle = 1.0
    Section 3.1 uses these chosen thresholds to select which tasks enter the suite. They shape task difficulty and composition but are not fitted to a particular model result.
assumptions (5)
  • domain assumption Internal usage taxonomies faithfully represent real coding-agent request distributions.
    Section 2 states task mixes are matched to aggregate distributions of real requests, but the taxonomies and data are not disclosed or externally validated.
  • domain assumption Colloquial rewriting of source artifacts makes prompts unsearchable.
    Section 2 asserts no prompt is recoverable by web search. No retrieval test, corpus-overlap check, or leakage audit is reported.
  • domain assumption Post-episode evaluation isolation holds in the sandbox for all tasks.
    Section 4 relies on the Dockerfile building only the agent-visible workspace and grading assets being introduced after completion. The enforcement across 260 tasks is assumed.
  • domain assumption Model-judge bias does not dominate Web, Office, or Code-reference scores.
    Sections 4 and 7 acknowledge an unquantified model-judge bias risk. The headline Code and Security metrics are deterministic, but Web and Office partly depend on LLM/VLM judges.
  • domain assumption Task freshness at authoring time means source artifacts are absent from pretraining corpora.
    Section 4 asserts tasks are built from content excluded from model-pretraining corpora before release. No membership or corpus-overlap audit is supplied.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction." pith.science (2026). https://pith.science/paper/SZBYRJKI

@misc{pith2026260720911,
  author       = {Pith},
  title        = {Pith review of: Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SZBYRJKI}},
  note         = {Machine review of arXiv:2607.20911}
}
read the original abstract

We introduce Tencent WorkBuddy Bench, a multi-domain evaluation suite for coding agents; this report documents its construction methodology, scoring protocol, and a cross-model leaderboard. At its core is a unified evaluation framework for constructing and running distribution-informed coding-agent tasks across four work domains - Code, Web, Office, and Security. Rather than adapting public issue text, every task is reverse-engineered from a real commit, pull request, or business scenario and rewritten as a short, colloquial, role-played request, so that a task's prompt is not recoverable by web-searching the underlying issue, pull request, or commit thread. Because the dataset is released openly - task directories, environment images, evaluation harness, tests, and reference solutions - contamination resistance rests on this construction together with dataset versioning rather than on secrecy. The four subsets - repository-level engineering, front-end development, office and business workflows, and red-/blue-team security - probe complementary facets of real work, each with its own verification style. All are packaged in a uniform task-directory format and run, under a uniform and reproducible protocol, on two agent harnesses (CodeBuddy Code and Claude Code); the full open release makes the benchmark reproducible end to end and directly auditable, since any third party can re-run each task and inspect its content. Because each subset uses a different scoring instrument, scores are not comparable across subsets and the suite reports no suite-wide average. We report a cross-model leaderboard across several model families.

Figures

Figures reproduced from arXiv: 2607.20911 by the authors.

Figure 1
Figure 1. Tencent WorkBuddy Bench at a glance. Real commits, pull requests, office workflows, and security cases are reverse-engineered into colloquial, role-played requests, with task distributions matched to real usage (left); the four tracks – Code, Web, Office, and Security – share one open task-directory format (center); every task is scored in an isolated sandbox under two agent harnesses (right). approach, drawing task… view at source ↗
Figure 2
Figure 2. Code and Web task composition. (a) Six Code usage domains merged from 18 fine-grained categories; bug fixing accounts for 10 of 80 tasks. (b) Seven Web task categories. (c) Six Web lifecycle modes; From Scratch accounts for 35 of 70 tasks. climb the L-ladder, a repository-complexity scale running from L2 (small, few modules) to L5 (large multi-module codebases) [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Code task and evaluation workflow. The coding agent reads a natural-language request, explores the repository, and emits a patch (left); the patch is graded by hidden unit tests and, as a diagnostic reference, by a rubric-weighted LLM judge (center). The headline Code metric is the hidden-test score; the LLM-judge reading and its blend are reported as reference values only and never enter the headline metric (right)… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Web task and evaluation workflow, from query and agent rollout to the delivered artifact (left), through rule, LLM/VLM, and agent judges over extracted evidence (center), to rubric-item checklist scoring (right), combining deterministic checks, semantic and visual judg…
Figure 5
Figure 5. Figure 5: Composition of the 50-task Office release by construction route and calibrated difficulty. 3.3 Office The Office subset tests whether an agent can complete a natural-language work request in a local workspace containing mixed-format files. Inputs include spreadsheets, …
Figure 6
Figure 6. Figure 6: Office coverage across task type, diagnostic scenario, output family, and evaluation mechanism, arranged as a single row of four bar charts. Output families and mechanisms use multi-label counts; scenario groups describe benchmark coverage and are not estimates of prod…
Figure 7
Figure 7. Figure 7: Office task, evaluation, and scoring flow. The agent acts on the workspace and leaves a final state (left); deterministic rule checks evaluate the verifiable workspace state while the LLM Judge evaluates only fixed post-task evidence (center); each task combines the tw…
Figure 8
Figure 8. Figure 8: WorkBuddy Bench Security overview. Tasks are built from real, historical vulnerabilities and authored scenarios into reproducible, self-contained cases (left); they span six red- and blue-team task types – whitebox source audit, blackbox binary exploitation, web exploi…
Figure 9
Figure 9. Figure 9: Office results by difficulty and task type. (a) Mean Office score by calibrated difficulty within each harness, averaged over all models with scored runs in that panel. (b) Equal-weight task average for the seven diagnostic task types represented by at least four tasks…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 6 canonical work pages

  1. [6]

    Harbor: A framework for evaluating and optimizing agents and models in container environments

    Harbor Framework Team. Harbor: A framework for evaluating and optimizing agents and models in container environments. GitHub repository, Laude Institute, 2026. URLhttps: //github.com/laude-institute/harbor. DOI: 10.5281/zenodo.20953922. 28 TENCENT WORKBUDDY BENCHTechnical Report·2026

    doi:10.5281/zenodo.20953922 2026

  2. [1]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world github issues? InInternational Conference on Learning Representations (ICLR), 2024. URLhttps://arxiv. org/abs/2310.06770

    arXiv 2024

  3. [2]

    Introducing SWE-bench verified

    OpenAI. Introducing SWE-bench verified. OpenAI blog, 2024. URLhttps://openai.com/ index/introducing-swe-bench-verified/

    work page 2024

  4. [3]

    Design2code: How far are we from automating front-end engineering? InProceedings of the Association for Computational Linguistics (ACL), 2024

    Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. Design2code: How far are we from automating front-end engineering? InProceedings of the Association for Computational Linguistics (ACL), 2024. URLhttps://arxiv.org/abs/2403.03163

    arXiv 2024

  5. [4]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. InInternational Conference on Learning Representations (ICLR), 2024. URLhttps://arxiv.org/abs/2307.13854

    arXiv 2024

  6. [5]

    How we compare model quality in Cursor

    Cursor. How we compare model quality in Cursor. Cursor blog, 2026. URLhttps://cursor. com/blog/cursorbench

    work page 2026

  7. [7]

    Chiu, Claire Cardie, Matthias Gallé, and Alexander M

    Wenting Zhao, Nan Jiang, Celine Lee, Justin T . Chiu, Claire Cardie, Matthias Gallé, and Alexander M. Rush. Commit0: Library generation from scratch, 2024. URLhttps://arxiv. org/abs/2412.01769

    arXiv 2024

  8. [8]

    LiveCodeBench: Holistic and contami- nation free evaluation of large language models for code, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contami- nation free evaluation of large language models for code, 2024. URLhttps://arxiv.org/ abs/2403.07974

    arXiv 2024

Show all 25 references
  1. [9]

    Tianyang Liu, Canwen Xu, and Julian J. McAuley. RepoBench: Benchmarking repository- level code auto-completion systems. InInternational Conference on Learning Representa- tions (ICLR), 2024. URLhttps://arxiv.org/abs/2306.03091

    2024 arXiv

  2. [10]

    Aider polyglot benchmark

    Paul Gauthier. Aider polyglot benchmark. Aider documentation, 2024. URLhttps://aider. chat/docs/leaderboards/

    2024

  3. [11]

    Terminal-bench: A benchmark for ai agents in terminal environ- ments

    Terminal-Bench Team. Terminal-bench: A benchmark for ai agents in terminal environ- ments. Project website, 2024. URLhttps://www.tbench.ai/

    2024

  4. [12]

    Vibe Code Bench: Evaluating AI models on end-to-end web application development

    Hung Tran, Langston Nashold, Rayan Krishnan, Antoine Bigeard, and Alex Gu. Vibe Code Bench: Evaluating AI models on end-to-end web application development. InACM Con- ference on AI and Agentic Systems (ACM CAIS), 2026. doi:10.1145/3786335.3813180. URL https://arxiv.org/abs/2603.04601

    2026

  5. [13]

    Jingyu Xiao, Yuxuan Wan, Yintong Huo, Zixin Wang, Xinyi Xu, Wenxuan Wang, Zhiyao Xu, Yuhang Wang, and Michael R. Lyu. Interaction2Code: Benchmarking MLLM-based interac- tive webpage code generation from interactive prototyping, 2024. URLhttps://arxiv. org/abs/2411.03292

    2024

  6. [14]

    FrontendBench: A benchmark for evaluating llms on front-end development via automatic evaluation, 2025

    Hongda Zhu, Yiwen Zhang, Bing Zhao, Jingzhe Ding, Siyao Liu, Tong Liu, Dandan Wang, Yanan Liu, and Zhaojian Li. FrontendBench: A benchmark for evaluating llms on front-end development via automatic evaluation, 2025. URLhttps://arxiv.org/abs/2506.13832

    2025 arXiv

  7. [15]

    VisualWebArena: Eval- uating multimodal agents on realistic visual web tasks

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Gra- ham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. VisualWebArena: Eval- uating multimodal agents on realistic visual web tasks. InProceedings of the Association for Computati...

    2024 arXiv

  8. [16]

    Workspace-Bench 1.0: Benchmarking AI agents on workspace tasks with large- scale file dependencies, 2026

    Zirui Tang, Xuanhe Zhou, Yumou Liu, Linchun Li, Yukai Wu, Weizheng Wang, Hongzhang Huang, Wei Zhou, Jun Zhou, Jiachen Song, Shaoli Yu, Jinqi Wang, Zihang Zhou, Hongyi Zhou, Yuting Lv, Jinyang Li, Jiashuo Liu, Ruoyu Chen, Chunwei Liu, GuoLiang Li, Jihua Kang, and Fan Wu. Worksp...

    2026 arXiv

  9. [17]

    ClawsBench: Evaluating capability and safety of LLM produc- tivity agents in simulated workspaces, 2026

    Xiangyi Li, Kyoung Whan Choe, Yimin Liu, Xiaokun Chen, Chujun Tao, Bingran You, Wenbo Chen, Zonglin Di, Jiankai Sun, Shenghan Zheng, Jiajun Bao, Yuanli Wang, Weixiang Yan, Yiyuan Li, and Han-chung Lee. ClawsBench: Evaluating capability and safety of LLM produc- tivity agents i...

    2026 arXiv

  10. [18]

    OdysseyBench: Evaluating LLM agents on long-horizon complex office application workflows, 2025

    Weixuan Wang, Dongge Han, Daniel Madrigal Diaz, Jin Xu, Victor Rühle, and Saravan Raj- mohan. OdysseyBench: Evaluating LLM agents on long-horizon complex office application workflows, 2025. URLhttps://arxiv.org/abs/2508.09124

    2025 arXiv

  11. [19]

    SpreadsheetBench 2: Evaluating agents on end-to-end business spreadsheet workflows, 2026

    Jian Zhu, Yuzheng Zhang, Zeyao Ma, Bohan Zhang, Armin Schoepf, Daniel Woloch, Pe- ter Yiliu Wang, Guangyu Robert Yang, Samuel Jacob, Siddharth Nagisetty, Abhiram Chundru, Jean Lin, Spencer Mateega, and Jing Zhang. SpreadsheetBench 2: Evaluating agents on end-to-end business sp...

    2026

  12. [20]

    Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W

    Andy K. Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W. Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, et al. Cybench: A framework for evaluat- ing cybersecurity capabilities and risks of language models. InInternational Conference on Lear...

    2025 arXiv

  13. [21]

    NYU CTF Bench: A scalable open-source benchmark dataset for evaluating LLMs in offensive security

    Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khor- rami, Ramesh Karri, and Muhammad Shafique. NYU CTF Bench: A scalable open-source benchmark dataset for ev...

    2024 arXiv

  14. [22]

    InterCode: Standard- izing and benchmarking interactive coding with execution feedback

    John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. InterCode: Standard- izing and benchmarking interactive coding with execution feedback. InAdvances in Neu- ral Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023. URL https://arxiv.org...

    2023 arXiv

  15. [23]

    CVE-Bench: A benchmark for AI agents’ ability to exploit real-world web application vulnerabilities

    Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, Jet Geronimo, Avi Dhir, Sudhit Rao, Kaicheng Yu, Twm Stone, and Daniel Kang. CVE-Bench: A benchmark for AI agents’ ability to exploit real-worl...

    2025 arXiv

  16. [24]

    CyberSecEval 2: A wide-ranging cybersecurity evaluation suite for large language models, 2024

    Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, David Molnar, Spencer Whitman, and Joshua Saxe. CyberSecEval 2: A wide-ranging cybersecurity evaluation suite for large langua...

    2024 arXiv

  17. [25]

    CyberSecEval 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models, 2024

    Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding, Vlad Ionescu, Yue Li, and Joshua Saxe. CyberSecEval 3: Advancing the evaluation of cybersecurity risks and capabilities...

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.