{"id":"19a2ec20-6d33-47d5-b934-d5ea3e2c3d73","arxiv_id":"2607.14989","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"OmniaBench introduces a broad 1,431-task agent benchmark covering 354 domains and reports that frontier models solve only about 58% of its 644-task challenging subset.","lead":"OmniaBench is a new benchmark that tests AI agents across 1,431 tasks spanning consumer, business, and workplace scenarios, with a hard subset of 644 tasks. It reports that even the best current models, Claude-Sonnet-5 and GPT-5.6-Sol, solve only about 58% of the hard tasks, and it scores agents along ten capability dimensions to show where they fail.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's Overall Pass@1 values (58.54/57.14) appear in Table 4's UserTurns column, not the OverallPass@1 column, which lists 64.16/38.73.","rationale":"The reader identified the weakest assumption as LLM-judge reliability and task solvability, with moderate Fleiss kappa. While that is a legitimate limitation (and the paper itself flags that the agreement does not rule out systematic bias), my stress-test pass found a more direct and checkable problem: the paper's headline numbers do not match its own data table. The abstract claims Overall Pass@1 of 58.54 and 57.14 for Claude-Sonnet-5 and GPT-5.6-Sol, but Table 4's OverallPass@1 column lists 64.16 and 38.73 for those models, and the claimed values appear under UserTurns. A quick weighted recalculation from the route success rates and task counts shows GPT-5.6-Sol's overall would be about 49.7, not 57.14. This means the central 'frontier models only reach ~58%' claim is, at best, misreported and, at worst, incorrect for GPT-5.6-Sol. The benchmark may still be valuable, and the underlying data may be fine, but the paper must be corrected and the values reconciled. This keeps the reader's CONDITIONAL verdict: the benchmark's claims cannot be fully accepted until the dataset, evaluation code, and corrected numbers are released. I found no evidence of fabrication or ad hominem; the issue appears to be a concrete error in reporting. I disagree that judge robustness is the single most load-bearing concern, because the numerical inconsistency is more directly tied to the paper's central claim and is immediately testable.","tokens_in":22449,"tokens_out":10736,"duration_ms":93420,"concrete_test":"Recompute each model's Overall Pass@1 as (DAG×354 + DAG-S×200 + Solver×60 + Program×30)/644 using Table 4's route columns. If GPT-5.6-Sol's result is ≈49.7 rather than the reported 57.14, the abstract's headline number is wrong. Also cross-check the original PDF to determine whether the 'OverallPass@1' and 'UserTurns' column headers/labels were transposed; if the table as published contains this mismatch, the paper must be corrected before the headline claim is accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is an internal numerical inconsistency affecting the central claim. The abstract and Section 4.2 state that Claude-Sonnet-5 and GPT-5.6-Sol achieve Overall Pass@1 of only 58.54% and 57.14%. But Table 4's 'OverallPass@1' column shows 64.16 for Claude-Sonnet-5 and 38.73 for GPT-5.6-Sol, while the values 58.54 and 57.14 appear in the 'UserTurns' column. Recomputing the weighted overall from route scores and task counts (DAG 354, DAG-S 200, Solver 60, Program 30) gives ~58.5 for Claude (consistent with 58.54) but ~49.7 for GPT-5.6-Sol, not 57.14. The paper's central measure of frontier difficulty is therefore not reproducible from its own results table, independent of any judge-bias or solvability concerns.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"OmniaBench is a large-scale benchmark for general LLM agents, containing 1,431 tasks (644 in a challenging subset) built from a 90-level-1/354-level-2 domain taxonomy spanning ToC/ToB/ToE. Tasks are constructed through four routes (DAG, DAG-S, Solver, Program) in executable environments with explicit state, tools, and multi-turn interaction. Evaluation uses LLM-based user simulators and rubric/VerifyCode judges. The paper evaluates 22 closed- and open-source models and reports that frontier models such as Claude-Sonnet-5 and GPT-5.6-Sol achieve Overall Pass@1 of only 58.54% and 57.14%, with diagnostic analyses across capabilities, domains, interaction turns, and error types.","tokens_in":22704,"tokens_out":11019,"duration_ms":101148,"significance":"If the benchmark is valid, OmniaBench would be a valuable complement to existing single-ecosystem agent benchmarks: it provides broad domain coverage, executable stateful environments, a ten-dimension capability taxonomy, eight difficulty factors, and a public code/data release. The paper is unusually transparent in reporting judge-robustness statistics (ICC(A,1)=0.780, Fleiss kappa=0.573, NMPD), user-simulator substitution results, and Pass@k/Pass^k reproducibility. These strengths make the benchmark potentially useful for diagnostic evaluation. However, the validity of the leaderboard and capability profiles depends on the correctness of LLM-generated tasks and the reliability of an LLM rubric judge, and the paper's own numbers contain an internal inconsistency that undermines the headline difficulty claim.","major_comments":[{"comment":"The headline difficulty claim is not reproducible from the paper's own results. The abstract and §4.2 state that Claude-Sonnet-5 and GPT-5.6-Sol achieve Overall Pass@1 of 58.54% and 57.14%, but Table 4's OverallPass@1 column lists 64.16 and 38.73 for these models, while 58.54 and 57.14 appear in the UserTurns column. Recomputing the exact weighted average defined in §4.1 from the route columns and the task counts (354 DAG, 200 DAG-S, 60 Solver, 30 Program) gives 54.8% for Claude-Sonnet-5 and 49.7% for GPT-5.6-Sol, using the printed Solver values 9.97 and 7.52. No reading of the table yields both headline numbers. Because the 'substantial challenge to frontier models' claim is the paper's central result, the table, abstract, and §4.2 must be reconciled and re-verified.","section":"Table 4 / §4.2 / Abstract"},{"comment":"The judge-robustness study is transparent, but the conclusions drawn from it are stronger than the numbers support. Fleiss' kappa for item-level pass/fail decisions is 0.573 (moderate), trajectory-level ICC(A,1)=0.780, 9.3% of items are 4-vs-1 splits, and 4.4% are 3-vs-2 splits. The paper itself notes (A.1) that this agreement 'does not by itself establish the correctness of the judgments or rule out systematic biases shared by multiple judge models.' Because the leaderboard and all capability profiles are produced by this judge model, the claim that OmniaBench provides a valid or diagnostic measure of capability boundaries requires either a human-validated gold-standard sample or a downstream validity argument. At minimum, report confidence intervals for the reported scores and treat the moderate item-level agreement as a first-order caveat in the main text, not only in the appendix.","section":"Appendix A.1"},{"comment":"Task solvability is a load-bearing assumption that is not directly verified. Environments and tasks are LLM-synthesized and then human-curated, but the paper reports no inter-annotator agreement, no count of tasks rejected or repaired during curation, and no independent human solvability audit of the final 644 tasks. If some tasks are subtly unsolvable or contain hidden contradictions, the reported pass rates measure task artifacts rather than agent capability. Please provide: (i) human-annotation statistics for the curation step; (ii) a solvability audit in which gold/reference trajectories are executed successfully in the environments; and (iii) a statement of how many tasks were removed or revised and why. This is needed to separate genuine difficulty from construction noise.","section":"§3.3 Data Construction"}],"minor_comments":[{"comment":"Numeric columns run together (e.g., '57.349.97', '60.5060.68'), making the table hard to read and likely contributing to the inconsistency above. Please reformat with clear column separators and verify all entries.","section":"Table 4"},{"comment":"The Program route entries for '# Items' and 'Item points' appear as '110.0' and '123.6' in the current text, probably due to missing delimiters; the intended values are presumably '1' and '1' (with Overall values '10.0' and '23.6'). Reformat and reconcile.","section":"Tables 3 and 9"},{"comment":"The sentence 'even the frontier models ... attains' should be 'attain' for subject-verb agreement.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The Table 4 inconsistency is serious and should be checked against raw evaluation logs before publication; the abstract, §4.2, and Table 4 cannot all be correct. I recommend the editor ask for a corrected machine-readable results table and a detailed reconciliation of the Overall Pass@1 computation. The moderate LLM-judge agreement is honestly reported, but the authors should be pushed to provide a human-annotated gold-standard sample and a solvability audit. The central benchmark contribution is potentially valuable, so I am not recommending rejection, but the revision must be substantive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. OmniaBench is a genuinely broad general-agent benchmark — 90 level-1 / 354 level-2 domains, four task-synthesis routes, a ten-dimensional capability taxonomy, and a thoughtful set of atomic difficulty factors. That is real engineering and real measurement thinking. But the headline numbers do not survive contact with the paper's own results table.\n\nTable 4's OverallPass@1 column shows 64.16 for Claude-Sonnet-5 and 38.73 for GPT-5.6-Sol. The abstract and Section 4.2 report 58.54 and 57.14 as Overall Pass@1 — those are the values in the UserTurns column. Recomputing the weighted overall from the route scores and task counts (354 DAG, 200 DAG-S, 60 Solver, 30 Program) gives about 54.8 and 49.7. None of the three sets match. The central claim about frontier difficulty is therefore not reproducible from the paper's own data. This is not a rounding issue; 64.16 cannot be obtained from route scores whose weighted average is below 55.\n\nWhat the paper does well: the taxonomy construction is concrete and grounded in app stores, PRDs, web retrieval, and human refinement. The four routes generate genuinely different task shapes. The capability and difficulty taxonomies are a useful organizing vocabulary. The appendix trajectory cases show real tasks with real rubrics. The paper is also unusually honest about judge robustness: ICC(A,1)=0.780 total score, Fleiss kappa=0.573 item decisions, and an explicit note that agreement doesn't establish correctness. That last sentence is more than most benchmark papers say.\n\nThe soft spots beyond the numbers: judge agreement is moderate, not high; the tasks are LLM-generated with human curation but no external human validation set; and the challenging set is 644 of 1,431 tasks, so the headline applies to the harder subset. Those are all addressable. The numerical inconsistency is the most serious because it undercuts the headline without any extra assumptions.\n\nWho this is for: researchers building or evaluating agent benchmarks will find the taxonomy and route design worth studying. The paper deserves a serious referee — it is not a desk reject — but the referee should demand a corrected results table, a precise definition of the Overall metric, and either external validation or a larger human-checked sample.\n\nRecommendation: send to peer review, but expect major revision on the reporting.","headline":"Broad and well-constructed agent benchmark, but the headline numbers are internally inconsistent: the abstract's 58.54/57.14 are actually the UserTurns column, and the OverallPass@1 column doesn't match the route-weighted arithmetic.","tokens_in":23270,"tokens_out":6054,"would_cite":false,"duration_ms":54721,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new 1,431-task benchmark spanning 90 domains holds today's strongest AI agents to about 58 percent first-attempt task-completion success.","keywords":["general AI agents","agent evaluation","capability taxonomy","task synthesis","tool use","multi-turn interaction","rubric-based evaluation","POMDP"],"falsifier":"Take a random sample of roughly fifty trajectories from the 644-task challenge set and have independent human experts re-score each rubric item under the same protocol, and separately have expert humans attempt to complete a sample of the tasks from scratch in the real environments. If experts fail to solve a substantial share of tasks, or if human judges reverse the LLM judge on more than the observed ~14 percent of contested items, then the 58 percent frontier ceiling and the capability profiles are measurement artifacts rather than agent capability boundaries.","tokens_in":22325,"feed_emoji":"🧪","tokens_out":8805,"duration_ms":76614,"temperature":0.7,"pith_summary":"The paper sets out to build a measurement instrument for general-purpose AI agents—programs that do not just answer questions but call tools, track changing state, and complete multi-step user goals—and claims that instrument is both broad and diagnostic. Broad means 1,431 executable tasks derived from a real-world taxonomy of 90 consumer, business, and employee-facing domains (354 sub-domains), covering single-turn and multi-turn interaction, file and code workspaces, and tool orchestration. Diagnostic means every task is labeled along ten capability dimensions and eight atomic difficulty factors, with success scored per rubric item and, where possible, by programmatic verification, so a model's failures can be traced to a specific ability rather than a single aggregate number. The load-bearing outcome is the difficulty measurement: the strongest tested models complete only about 58 percent of the harder 644-task subset on the first attempt (Pass@1), and error analysis attributes most failures to planning, constraint maintenance, and adaptive correction rather than raw tool invocation. A sympathetic reader would care because, if the benchmark is valid, it redraws the frontier of what current agents can and cannot do autonomously and points to which capabilities deserve the next round of engineering.","feed_headline":"Top AI agents clear just 58 percent of new benchmark","feed_subtitle":"OmniaBench spans 90 domains to trace agent failures to planning and constraint slips, not tool calls.","key_machinery":"The carrying object is the three-view taxonomy feeding an environment-and-task factory, plus the evaluation protocol that fixes what the scores mean. The domain taxonomy, assembled from app-store categories, product documents, industry resources, web retrieval, and human refinement, defines coverage; the capability taxonomy defines diagnosis by attaching up to ten ability labels (Task Understanding through Reliability & Safety) to every task; the eight atomic difficulty factors define why a task is hard (ambiguity, long-context evidence, progressive disclosure, multi-source inconsistency, and so on). Four synthesis routes convert these views into POMDP-style environments: DAG builds multi-tu","core_discovery":"OmniaBench's central claim is that general-agent competence can be measured across the breadth of real-world applications by combining three structured views—a domain taxonomy (90 level-1 and 354 level-2 domains spanning consumer, business, and employee-facing settings), a ten-dimensional capability taxonomy, and eight atomic difficulty factors—instantiated as 1,431 executable, stateful tasks built through four synthesis routes (DAG, DAG-S, Solver, Program). Under its trajectory-based protocol—all-or-nothing rubric items adjudicated by an LLM judge, plus a programmatic VerifyCode verifier for code tasks—the two strongest evaluated models, Claude-Sonnet-5 and GPT-5.6-Sol, achieve Overall Pass","pith_inferences":["The rubric-checklist-plus-verifier protocol is a reusable template: any group with a simulated environment could assemble a tailored agent benchmark from the same pipeline, so the construction method may prove more durable than the specific 1,431 tasks.","The eight atomic difficulty factors invite ablation pairs—two tasks identical except for one factor, such as progressive disclosure on or off—which would isolate each factor's contribution to model failure; the paper reports aggregate capability differences but stops short of per-factor isolation.","Because the challenging set deliberately samples each domain's longest tool chain, the headline 58 percent describes the hard tail; a reader should expect materially higher pass rates on the full set, and the difference between the two sets is itself a publishable measure of horizon sensitivity.","Around one in seven rubric-item decisions is non-unanimous across the five judge models tested; re-deriving the paper's error-mode percentages from unanimously agreed items only would show how much of the planning-dominance conclusion rides on contested judgments."],"forward_implications":["If the benchmark measures what it claims, the roughly 58 percent ceiling means no current agent can be trusted with fully autonomous completion of long, stateful, multi-tool tasks; on the hard subset, about half of tasks are only partially completed.","Because capability profiles diverge between models with similar overall scores, deployment decisions should be made per capability and per domain, not on a single leaderboard rank.","The error distribution—planning and decomposition at 36.7 percent, constraint violations at 16.5 percent, premature abandonment prominent among meta-cognitive errors—points model improvement at plan generation, constraint tracking, and recovery behavior rather than at function-calling syntax.","Domain-level rank reversals imply no single model dominates all 90 domains; specialized agent portfolios could beat any one general agent in a given application area.","The measured gap between Pass@k (success improves with restarts) and Pass^hat-k (success is not reproducible across runs) indicates that the immediate bottleneck is single-attempt reliability, not raw capability."],"fun_headline_variants":["New AI benchmark spans 90 domains, top models score 58%","AI agents fail a third of tasks in new diverse benchmark","OmniaBench: 1,431 tasks, 90 domains, top AI score just 58%","Planning and constraint slips drag down AI agent scores in new test","Broad new benchmark shows AI agents struggle on real-world tasks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole measurement stands on one premise: the 1,431 synthesized tasks are genuinely solvable as written, and the LLM rubric judge's pass/fail calls match what a careful human expert would decide—and the paper's own judge study finds only moderate item-level agreement (Fleiss kappa = 0.573), so a systematic judge bias or a batch of subtly unsolvable tasks would shift every reported score and capability profile.","fun_headline_variants_meta":{"raw":{"variants":["New AI benchmark spans 90 domains, top models score 58%","AI agents fail a third of tasks in new diverse benchmark","OmniaBench: 1,431 tasks, 90 domains, top AI score just 58%","Planning and constraint slips drag down AI agent scores in new test","Broad new benchmark shows AI agents struggle on real-world tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00097,"raw_usage":{"total_tokens":4001,"prompt_tokens":821,"completion_tokens":3180,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":3085}},"tokens_in":565,"tokens_out":3180,"duration_ms":18930,"temperature":1.0,"reasoning_tokens":3085,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T00:29:49.557251+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of roughly fifty trajectories from the 644-task challenge set and have independent human experts re-score each rubric item under the same protocol, and separately have expert humans attempt to complete a sample of the tasks from scratch in the real environments. If experts fail to solve a substantial share of tasks, or if human judges reverse the LLM judge on more than the observed ~14 percent of contested items, then the 58 percent frontier ceiling and the capability profiles are measurement artifacts rather than agent capability boundaries.","supporting_citations":[],"review_version":1}