{"id":"a0581c5c-6ffa-4f28-94b4-4fd3ac678a45","arxiv_id":"2607.20911","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Tencent WorkBuddy Bench is a 260-task, four-domain coding-agent benchmark whose colloquial prompts are reverse-engineered from real commits, CVEs, and business scenarios, released openly with a dual-harness cross-model leaderboard.","lead":"Tencent WorkBuddy Bench is a new open benchmark for coding agents, with 260 tasks across code, web, office, and security work. Each task is rewritten from a real commit, CVE, or business scenario into a colloquial request, so the benchmark is designed to resist web-searchable prompt leakage.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The contamination-resistance claim rests on an unverified rewriting-sufficiency assumption: Section 2 asserts prompts are unsearchable by construction, but no retrieval or corpus-overlap audit is reported, and the claimed fully open release has no URL, commit hash, or DOI in the manuscript.","rationale":"The reader's verdict is already CONDITIONAL, and it flags both the unverified representativeness of the internal usage taxonomies and the untested colloquial-rewriting sufficiency. My stress-test pass narrows the load-bearing concern to the same family but sharpens it: the rewriting-sufficiency assertion is not merely untested; it is the one claim on which the entire contamination-resistance value proposition rests, and the paper provides no audit evidence for it beyond the design description. I also elevate the open-release claim as a separate load-bearing prerequisite, since no URL, commit hash, or DOI appears anywhere in the manuscript, despite such identifiers appearing for the cited Harbor framework. Both conditions are checkable post-release, and both are prerequisites for the claimed auditable, contamination-resistant contribution. I do not find an internal inconsistency or an unsupported quantitative derivation: the admission gates (baseline <= 0.3, oracle >= 1.0) are explicit, the scoring formalisms are clearly specified, the limitations are candid, and the decision not to report a suite-wide average is internally coherent. The leaderboard's three-run averages lack error bars, but the paper does not claim statistical precision, and that is a reporting weakness rather than a central-claim failure. Because the concern is essentially the one the reader identified, the verdict stays CONDITIONAL: the paper should be accepted only once a working public artifact link and a retrieval audit are provided or confirmed.","tokens_in":24804,"tokens_out":1869,"duration_ms":17486,"concrete_test":"Once the dataset repository or DOI is provided, run a leakage/retrieval audit over the full instruction set: for each task's instruction.md, perform web-search queries and corpus-overlap comparisons against the underlying commit/PR/CVE sources (or, if sources are withheld, against a public GitHub code-index snapshot), measuring the fraction of tasks whose instruction text or a distinctive contiguous n-gram (e.g., a 8-token span) retrievably matches the source artifact. Separately, verify release completeness by attempting an end-to-end re-run of a stratified sample of tasks using only the public artifacts, confirming that every component listed in Table 2 is present and that a pinned commit hash or versioned DOI is provided.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The load-bearing claim, Section 2, is that no task's instruction text is recoverable by web-searching its underlying issue, pull-request, or commit thread, because tasks are reverse-engineered and rewritten into colloquial role-played requests. This is asserted by construction, but the paper reports no retrieval or corpus-overlap audit, so the claim is not demonstrated. The concrete failure mode: if an instruction retains distinctive identifiers, code symbols, function names, or problem-specific phrasing from the source commit, a near-verbatim web search could surface the source artifact, and a model trained on crawled Web data could recognize the task, defeating the benchmark's stated contamination resistance exactly where it matters. A second load-bearing claim, Section 2.1, asserts the suite is released fully open (task directories, environment images, evaluation harness, grading tests, reference solutions), but Table 2 lists components without any repository URL, commit hash, dataset DOI, or release identifier. The paper does give a GitHub URL and Zenodo DOI for the cited Harbor framework, so the absence of any artifact link for WorkBuddy Bench itself is conspicuous. If the release is not in fact complete and public, the reproducibility and auditability claims have no verifiable object. The paper's own Limitations section candidly concedes open-release contamination exposure and model-judge bias, but those are secondary; the two prerequisites for the benchmark's value proposition are the unaudited rewriting sufficiency and the unverifiable open release.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces Tencent WorkBuddy Bench, a 260-task benchmark (Code 80, Web 70, Office 50, Security 60) for coding agents. The main methodological proposal is that every task is reverse-engineered from a real upstream commit, CVE, or business scenario and rewritten as a short, colloquial, role-played instruction, so that the prompt text cannot be found by web-searching the underlying source thread; the suite is then released openly with task directories, images, harness, tests, and reference solutions, and contamination resistance rests on construction plus versioning. Each track has its own verifier (hidden tests, rubric judges, rule/Judge blend, deterministic scorer), scores are not aggregated across tracks by design, and results are reported for seven models under two harnesses. The paper also reports token/turn efficiency and per-category analyses.","tokens_in":1629,"tokens_out":1884,"duration_ms":98124,"significance":"The benchmark fills a real gap if its two load-bearing claims hold: construction-level resistance to web-searchable prompts and a fully open release enabling third-party audit. The paper is unusually candid, disclosing judge-bias risk, post-release contamination exposure, the modified-instruction cell, and single-language bias; the Security scorer is deterministic with anti-cheat layers, and the Office/Web scoring equations are explicit. However, the contamination-resistance claim is a design assertion without a retrieval audit, the open-release claim has no artifact identifier, and Figure 1 contradicts the stated no-overall-score principle. The contribution is therefore significant but conditional on verification and repair of these points.","major_comments":[{"comment":"Section 1 and Section 2 state repeatedly that scores are not comparable across subsets and that the suite reports no suite-wide average, yet Figure 1 displays an 'Overall Score' column with ranks (e.g., Claude Opus 4.8 at 75.0%). The listed values do not match any disclosed aggregation of Table 6 (the unweighted mean of that model's four Table 6 scores is 72.3), so the figure either contradicts the stated design principle or rests on an undocumented weighting. Please remove the column or provide the aggregation formula and justify its use.","section":"§1; Figure 1"},{"comment":"Section 2's 'Contamination-resistant task construction' paragraph asserts that no task instruction is recoverable by web-searching the underlying commit, issue, or PR thread, but this is argued by construction rather than demonstrated. The manuscript reports no retrieval or corpus-overlap audit, no search-query construction, and no n-gram or near-duplicate statistics, even though rewritten prompts can retain identifiers, code symbols, or distinctive phrasing from their source artifacts. Because this claim is load-bearing for the benchmark's central value, please add a reproducible retrieval audit over the released instruction texts against the source threads and a web corpus, and report overlap rates and retrieval recall.","section":"§2, 'Contamination-resistant task construction'"},{"comment":"Section 2.1 and Table 2 claim that the suite is released fully open — task directories, environment images, evaluation harness, grading tests, and reference solutions — and that any third party can re-run each task. However, the manuscript contains no repository URL, dataset DOI, commit hash, or release identifier for WorkBuddy Bench itself; the only artifact links given are for the Harbor framework (reference [6]). Without a persistent identifier and version, the auditability and reproducibility claims have no concrete object. Please provide the release URL/DOI, commit hash, and checksums for the task directories and images.","section":"§2.1; Table 2"},{"comment":"Section 2 says each subset's category mix, modes, roles, and difficulty are matched to the distribution of real requests using internal usage taxonomies, but neither the taxonomies nor the aggregation method is disclosed, and Section 3.3 explicitly states that the Office scenario groups 'describe benchmark coverage rather than estimate production request traffic.' This is a direct tension in the representativeness claim. Please either release the taxonomy and sampling/validation procedure or soften the distribution-matched framing to a coverage-design claim.","section":"§2; §3.3"},{"comment":"Table 6 and Section 5 report three-run means without per-run scores, variances, or confidence intervals, while making strong rank and harness-sensitivity claims (e.g., the Security mean absolute shift of 8.6 points and the reordering of GPT-5.5 from sixth to second). Given known run-to-run variability in agent evaluations, these claims need uncertainty quantification; at minimum, provide per-run scores or bootstrap confidence intervals. Please also state explicitly how Security refusals (e.g., Claude Opus 4.8's 13 under Claude Code) are counted in the reported reward.","section":"§5; Table 6"}],"minor_comments":[{"comment":"The Contributions section states the admission gate requires 'oracle reward ≥ 1.0,' while Section 3.1 says 'oracle reward = 1.0'; make the two statements consistent.","section":"§1; §3.1"},{"comment":"Figure 1's leaderboard includes Kimi K2.7 (Security '—'), but Table 6 omits Kimi and includes DeepSeek-V4-Flash; clarify which models are in the official leaderboard and why.","section":"Figure 1; Table 6"},{"comment":"The footnote to the Security turn statistics says they still use an earlier turn-counting convention; since the text compares Security with Code, Web, and Office, either recompute Security under the same convention or explicitly mark all Security efficiency figures as non-comparable.","section":"§5.1"},{"comment":"Equation (2) should state whether the configured penalties p_{t,i} are normalized so that the max(0, 1 − sum) expression has a consistent scale across tasks, and whether fatal failures also incur a configured penalty.","section":"§4, Eq. (2)"},{"comment":"The canary-string mechanism is mentioned only in passing; a short technical description of the format, insertion policy, and detection method would make the versioning claim concrete and auditable.","section":"§2, versioning paragraph"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is honest and well organized, but the gap between the headline claims and the verifiable artifacts is the main risk. I would require an artifact DOI/URL and a retrieval audit before acceptance, and the Figure 1 overall-score contradiction should be fixed in the same revision. The paper is essentially a technical report for a new benchmark, so the venue should confirm that benchmark papers are expected to include release artifacts in the manuscript itself; the absence of any WorkBuddy Bench artifact link is unusual and needs to be addressed editorially."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth reading. This is a benchmark construction report with an unusual amount of methodological candor. The core contribution is real: unifying four work domains (Code, Web, Office, Security) into one open, Harbor-style task format, with tasks reverse-engineered from real commits/CVEs/business scenarios and rewritten as colloquial role-played requests. That combination is new. The scoring design is sensible — no cross-subset average, deterministic hidden tests for Code, deterministic scorer for Security, LLM judgments kept out of the headline metrics where possible, and even the modified-instruction run and HY-3 first-party serving asymmetry disclosed. The admission gate (baseline ≤ 0.3, oracle ≥ 1.0) is a good hygiene step. Related-work positioning is honest, explicitly design-time rather than head-to-head. Credit is due.\n\nThe weak spots align with the stress-test note. The two load-bearing claims are not actually verified in the document. Section 2 asserts that no task's instruction text is recoverable by web-searching the source commit/PR thread, but no retrieval or corpus-overlap audit is reported. That is the difference between a design principle and a demonstrated property. Second, Section 2.1 says the suite is fully open, but the manuscript contains no dataset URL, commit hash, or DOI for the benchmark itself — only the Harbor citation has a link. For a paper whose entire value proposition is independent auditability, that absence is conspicuous. If the release artifacts are complete and the rewriting-sufficiency claim is backed by an audit, these are fixable in revision. If the paper asks the reader to take them on faith, that is the gap.\n\nA couple of additional smaller issues: the distribution-informed task mix depends on undisclosed internal taxonomies, and the Code track is Python-dominated (acknowledged). Leaderboard numbers have no error bars across the three runs. None of these are fatal; they are proportionate caveats. The circularity burden is low — the free parameters are per-task scoring weights, not fitted to produce the leaderboard.\n\nWho is this for? Anyone building or using coding-agent benchmarks will get value. It deserves a serious referee, but the referee should ask for the artifact link and a leakage audit before acceptance.","headline":"A genuinely useful, well-documented benchmark construction, but the two load-bearing claims — unsearchable prompts and an auditable open release — are asserted rather than demonstrated in the manuscript.","tokens_in":25751,"tokens_out":2056,"would_cite":true,"duration_ms":17379,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WorkBuddy Bench claims that reverse-engineering real commits into colloquial prompts makes a fully open benchmark resistant to web-search contamination.","keywords":["multi-domain coding-agent benchmark","contamination-resistant construction","role-played task rewriting","open benchmark auditability","repository-level software engineering","web front-end evaluation","office workflow evaluation","security agent evaluation"],"falsifier":"For each task in the released 260-task suite, take the instruction text and query a general web search engine, as well as a large code corpus, for the originating repository, commit hash, pull-request number, or CVE description; then count how many tasks return their source thread among the top results. If a substantial fraction do, the claim that prompts are unrecoverable by web-searching fails.","tokens_in":24597,"feed_emoji":"🧪","tokens_out":10014,"duration_ms":88056,"temperature":0.7,"pith_summary":"WorkBuddy Bench is an open, 260-task evaluation suite for coding agents that tries to solve the contamination problem at the point of task writing rather than by hiding the dataset. Each task is reverse-engineered from a real commit, pull request, CVE, or business scenario and rewritten as a short, colloquial, deliberately underspecified request, so a web search for the underlying issue or PR does not recover the prompt. The suite spans four work domains, Code, Web, Office, and Security, under one task-directory format and one sandboxed execution harness, with each domain scored by its own instrument and no suite-wide average. The full release includes task directories, environment images, grading tests, and reference solutions, so any third party can re-run a task and audit the result. If the construction claim holds, the benchmark offers a reproducible, auditable alternative to both memorization-prone public suites and closed vendor benchmarks.","feed_headline":"Open benchmark rewrites real commits into unsearchable tasks","feed_subtitle":"Four work domains share one open task format and prompts engineered so web search cannot find their source.","key_machinery":"The mechanism that carries the argument is a construction protocol rather than a single theorem: each task is anchored to a real upstream artifact, such as a historical commit, pull request, or CVE, or to an abstracted business scenario, and then rewritten as a short, colloquial, deliberately underspecified request voiced through a requester persona, with the root cause and reference diff withheld from the agent. This is what makes the prompt unavailable to web search by construction. The second carrier is a Harbor-style task-directory format in which the agent-visible Docker workspace contains only the repository or business files, while grading assets live in a separate tests/ directory and are introduced only after the episode ends. Code tasks additionally pass an oracle-gated admission check, with baseline reward $\\le 0.3$ and oracle reward $= 1.0$, before release.","core_discovery":"The paper's central claim is that a benchmark can be both fully open and resistant to the contamination path that matters most, web-searchable prompts, if the prompts are constructed rather than collected. Every task is anchored to a concrete origin, a historical commit or pull request in an open-source repository, a historical CVE, or an abstracted business scenario, and rewritten as a short, colloquial, role-played request that withholds the root cause, the reference diff, and any solution-hinting framing. Grading assets are introduced only after the agent has finished acting, and the released package contains everything needed to reproduce and audit a run. The paper also claims that the distribution of tasks, in categories, modes, roles, and difficulty, is matched to internal usage taxonomies of real requests, and that this is done without using or releasing any raw user data. Because the four subsets use different scoring instruments, the suite reports per-subset scores and deliberately no suite-wide average.","pith_inferences":["My inference: the reverse-engineer-and-rewrite recipe is domain-neutral, so the same template could be carried to notebook data science, DevOps runbooks, or design systems; the durable contribution may be the construction format rather than the four current domains.","My inference: the central contamination claim would become measurable if the project ran a retrieval audit at each release, searching each prompt against its source commit, PR, or CVE corpus, and published the match rate.","My inference: because the internal usage taxonomies behind the task mix are not disclosed, representativeness is currently an assumption; publishing those taxonomies in aggregate form would turn it into a testable claim.","My inference: the observed harness-dependent reordering suggests benchmark consumers should treat a leaderboard as bound to the harness, not as a model-quality fact."],"forward_implications":["Any third party can re-run every task and inspect the grading tests and reference solutions, so reported scores are auditable rather than trusted from a closed vendor.","Because prompts are written from reverse-engineered artifacts rather than copied from issue text, memorizing the underlying commit or pull-request thread does not by itself hand the agent the answer at release time.","Rankings depend on the harness: the paper's leaderboard shows model order shifting between CodeBuddy Code and Claude Code, most strongly on Security, so a single overall ranking would not be meaningful.","The four subsets are separate instruments with different scoring rules, so scores should be compared only within a subset; the suite intentionally reports no suite-wide average.","Code's difficulty concentrates in bug-fix and api-contract tasks, where colloquial requests must be grounded in the right lines of a repository, a finding consistent with the paper's view that navigation and understanding, not synthesis, limit current models."],"supporting_citations":[{"why":"Supplies the dominant public-suite format, GitHub issue text plus hidden tests, that the construction protocol is designed to avoid.","marker":"[1]"},{"why":"The verified SWE-bench subset, used alongside [1] as the reference point for memorization-prone public benchmarks.","marker":"[2]"},{"why":"Represents crawlable design-to-code generation benchmarks that motivate Web's artifact-not-chat contract and search-resistant construction.","marker":"[3]"},{"why":"Represents web-agent benchmarks operating on crawlable live sites, the contrast for Web's offline, artifact-based verification.","marker":"[4]"},{"why":"The closed production benchmark whose task distribution cannot be audited, the contrast that motivates the fully open release.","marker":"[5]"},{"why":"Supplies the Harbor task-directory convention that the suite adapts to keep grading assets outside the agent-visible workspace.","marker":"[6]"}],"fun_headline_variants":["Real commits become unsearchable role-played tasks","Open benchmark hides real work in unsearchable prompts","Benchmark rewrites real commits into unsearchable requests","Coding-agent benchmark: real tasks, disguised prompts, open audit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that colloquially rewriting a real commit or CVE makes each prompt unrecoverable by web-searching the source thread, a property the paper asserts by construction without a retrieval or corpus-overlap audit.","fun_headline_variants_meta":{"raw":{"variants":["Real commits become unsearchable role-played tasks","Open benchmark hides real work in unsearchable prompts","Benchmark rewrites real commits into unsearchable requests","Coding-agent benchmark: real tasks, disguised prompts, open audit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000906,"raw_usage":{"total_tokens":3935,"prompt_tokens":1019,"completion_tokens":2916,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":2849}},"tokens_in":635,"tokens_out":2916,"duration_ms":21138,"temperature":1.0,"reasoning_tokens":2849,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:31:08.260108+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For each task in the released 260-task suite, take the instruction text and query a general web search engine, as well as a large code corpus, for the originating repository, commit hash, pull-request number, or CVE description; then count how many tasks return their source thread among the top results. If a substantial fraction do, the claim that prompts are unrecoverable by web-searching fails.","supporting_citations":[{"cited_title":"Introducing SWE-bench verified","cited_arxiv_id":null,"evidence_quote":"The verified SWE-bench subset, used alongside [1] as the reference point for memorization-prone public benchmarks."},{"cited_title":"How we compare model quality in Cursor","cited_arxiv_id":null,"evidence_quote":"The closed production benchmark whose task distribution cannot be audited, the contrast that motivates the fully open release."}],"review_version":2}