{"id":"ecba533d-5f1e-49c5-8244-e21530f85422","arxiv_id":"2608.02441","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"ACWorld provides an auditable two-sided commerce benchmark in which process-level traces, not final states, are required to detect LLM agent errors (65.9–85.6% on the capability track; 56.1–91.4% on the large-catalog track).","lead":"ACWorld is a simulated marketplace where AI buyer and merchant agents transact through a validation protocol that logs every decision, authorization, and state change. On 260 tasks across ten LLMs, it shows that judging only the final outcome misses errors that the logged process catches.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim of evaluating independent Buyer and Merchant policies together is not demonstrated: benchmark fixes one side per task, and the multiagent study uses a single model family.","rationale":"The reader's weakest_assumption focuses on author-defined scoring predicates lacking external audit. That is a valid concern about benchmark validity, but it does not touch the paper's core novelty: the environment's ability to evaluate independently controlled Buyer and Merchant policies in a shared market. The experimental evidence does not exercise that property: every benchmark task holds the counterparty fixed, and the multiagent study uses one model family. The paper's own limitation statement (§8) admits this. My concern is therefore more load-bearing because it directly questions whether the central contribution has been demonstrated at all. A single concrete experiment with two distinct model families would settle it. Since the issue is a missing demonstration rather than a demonstrated failure, the verdict should remain conditional: accept only if the environment is shown to support genuine two-sided independent-policy interaction.","tokens_in":26281,"tokens_out":3674,"duration_ms":42467,"concrete_test":"Run the 5×5 multiagent study with two distinct model families (e.g., GPT-5.6 Sol as Buyers and Claude Sonnet 5 as Merchants), repeated over multiple seeds, and verify that VCP validation, atomic World commits, state reconstruction, and per-actor attribution all behave correctly. If the environment correctly settles a shared market under two independent policies, the central claim is supported; if not, the claimed two-sided evaluation capability is unverified.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim (§1, Abstract) is that ACWorld 'evaluates independently controlled Buyer and Merchant policies together' in a shared market. The benchmark protocol, however, evaluates only one role per task: 'Each task assigns one Buyer or Merchant to the evaluated model while every other agent follows a fixed policy' (§4). Thus the 2,000 and 600 runs measure single-policy performance against deterministic scripts, not two independently controlled policies interacting. The multiagent study (§5.4, Appendix A) fixes a 5×5 market but uses one model family and one seed, and the paper itself notes 'One market, model family, and seed cannot support strategy comparisons.' Section 8 concedes: 'Each task evaluates one policy against deterministic counterparties. Neither the scores nor the fixed 5×5 study measure strategy in an open market.' This is a gap between the claimed contribution and the supplied evidence. The environment may well support independent two-sided evaluation, but no reported experiment actually demonstrates it. If the environment fails under two genuinely independent policies — e.g., due to hidden state leakage, authority attribution errors, or interaction deadlocks — the primary novelty would be unsupported, regardless of how well the current benchmark scores these models.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Agentic Commerce World (ACWorld), an environment and benchmark for evaluating LLM-based Buyer and Merchant agents in a shared, persistent commerce market. The core architectural idea is the Vibe Commerce Protocol (VCP), which separates an agent's proposal from the authority to change shared state: the Commerce Intelligence Platform validates each typed action, and only then does the World commit the authorized effect. Every decision is stored as a linked record (actor, decision, validation, execution evidence, state delta), and traces can be deterministically reconstructed and rescored from the initial state and ordered commits. The benchmark has a 200-task capability-coverage track and a 60-task large-catalog track over 785,022 listings. Ten models are evaluated, with mean scores 65.9–85.6% and 56.1–91.4% on the two tracks. The paper also reports a failure-attribution analysis, a restricted-view collision analysis (99/861 runs without full credit share a final state with a full-credit run), a multiagent market study, and a set of validity gates: 200 reference episodes pass, 95 targeted mutations change intended checks, all 2,000 traces reconstruct, and rescoring is deterministic. The authors explicitly limit the scope: each benchmark task evaluates one policy against deterministic counterparties, and the fixed 5×5 study does not support strategy comparisons.","tokens_in":26577,"tokens_out":3614,"duration_ms":39506,"significance":"If the claims hold, the paper makes a useful contribution: it provides a reproducible, auditable environment for agentic commerce, with process-level evidence instead of final-outcome-only scoring, and it backs the environment with unusually thorough internal validation. The strongest parts are the machine-checkable verification infrastructure: all reference episodes pass, targeted mutations affect intended predicates, state reconstruction and deterministic rescoring are verified on all 2,000 traces, and the equal-weight sensitivity analysis (Table 20) checks the hand-set predicate weights. These are concrete, reproducible guarantees that distinguish the work from benchmarks that rely on LLM judges or loose human evaluation. The restricted-view collision finding (Section 5.6) is a well-posed information-insufficiency result, not an overclaimed error-rate statement. The main weakness is that the paper's headline contribution — 'independently controlled Buyer and Merchant policies together' — is not actually exercised by the benchmark protocol, and the one multiagent study uses a single model family and seed. This is a gap between the central claim and the supplied evidence, and it n","major_comments":[{"comment":"The central claim that ACWorld 'evaluates independently controlled Buyer and Merchant policies together' is not demonstrated by the reported experiments. The benchmark protocol in §4 states that 'Each task assigns one Buyer or Merchant to the evaluated model while every other agent follows a fixed policy.' Thus the 2,000 and 600 runs measure single-policy performance against deterministic scripts, not two independently controlled policies interacting. The multiagent study in §5.4 uses ten agents in a 5×5 market but with one model family and one seed, and §8 admits that 'Neither the scores nor the fixed 5×5 study measure strategy in an open market.' The environment may well support independent two-sided evaluation, but no experiment actually shows two independently controlled policies jointly producing a shared outcome. This is load-bearing because the paper's contribution (ii) and the ab","section":"§1, §4, §5.4, §8"},{"comment":"The benchmark's validity rests on the author-defined task predicates and reference policies. Equation (3) aggregates declared weights w_i over Boolean/fractional predicates c_i, and the entire failure analysis treats these predicates as ground truth. The paper's internal validation (reference episodes pass, mutations lower intended checks, deterministic rescoring) demonstrates self-consistency but not that the predicates correctly operationalize commercial intent. The paper itself notes in §8 that 'Reconstruction establishes consistency under the declared transition and scorer, not their independent semantic correctness' and that catalogs have not received an independent audit. Given that the benchmark is a central deliverable, the absence of any external check — e.g., independent human annotation of a sample of predicate verdicts, or a comparison with a practitioner-defined scoring rubr","section":"§4.2, §8"},{"comment":"The multiagent environment study is too thinly specified to support the 'independent policies in a shared market' narrative. The 5×5 study uses one model family, one seed, and a single deterministic clearing rule; the sparse-world probe in Appendix K uses scripted agents and excludes model inference, VCP routing, and contention. The paper appropriately disclaims that 'One market, model family, and seed cannot support strategy comparisons,' but this disclaimer also means the study provides only existence evidence that the environment can execute a multiagent market. Concretely, I would like to see either (i) repeated runs over multiple seeds with the current model, and (ii) at least one configuration where the Buyer and Merchant roles are played by different model families. Without this, the unique two-sided claim remains an architectural promise rather than an empirically supported prope","section":"§5.4, Appendix K"}],"minor_comments":[{"comment":"The case study gives a clear example of partial credit, but the weight values (0.214, 0.286, etc.) appear without explaining how the four predicates sum to 1.0. A one-line note that weights are normalized per task would help.","section":"§5.5.2 / Figure 9"},{"comment":"Finding 3 is explicitly qualitative and based on 'qualitative inspection.' It is appropriately labeled as an observation, but the text could more clearly state that no inter-annotator agreement or statistical test was used, to avoid readers treating it as a measured model property.","section":"§5.6, Finding 3"},{"comment":"The skill manifests are counted as 11 Buyer and 25 Merchant in Table 3, and Table 7 lists the same names grouped by function. It would improve readability to note that a skill manifest can have multiple primary functions, or that the counts refer to manifests rather than individual skills, since the grouping in Table 7 does not map one-to-one to the numbers.","section":"Table 3 / Table 7"},{"comment":"Table 11 shows that 17 of 95 targeted mutations are not detectable from final state, but the main text's Finding 1 uses the empirical collision rate of 11.5% from Table 5. It would be helpful to connect these two numbers explicitly: the mutation result is a designed lower bound on blind spots, while the collision result measures observed frequency in the model panel.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically solid in its internal validation and the environment design is genuinely useful. My main concern is that the paper's central framing oversells two-sided independent evaluation relative to what is actually run. This is fixable by either a modest additional experiment or a careful rewording of the contribution. I do not see a fatal flaw in the environment itself, so I am not recommending rejection. The paper would also benefit from a clear statement that the benchmark's scores are not a validated measure of real-world commercial competence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline should be: the engineering and validation are for real, but the paper sells its central evaluation claim harder than the experiments support. That said, the limitation is openly disclosed in §8, so it's a framing problem, not a hidden one.\n\nWhat is new: ACWorld separates agent decisions from authority to mutate shared state via VCP, and scores on verified intermediate process evidence rather than final outcome. The 99/861 collision count under final-state view is a genuine, clean demonstration that final state alone can miss evaluated errors. The validation work is unusually thorough for a benchmark paper: 200 reference episodes at full score, 95 targeted mutations that flip intended predicates, 374 large-catalog mutations, full trace reconstruction, deterministic rescoring. That is real evidence of internal consistency.\n\nThe soft spots are in proportion. First, the abstract and §1 say ACWorld 'evaluates independently controlled Buyer and Merchant policies together.' In practice, each benchmark task runs exactly one evaluated policy against scripted counterparties. The 5×5 multiagent study uses one model family and one seed, and the authors themselves say it cannot support strategy comparisons. So the unique selling point—independent two-sided policy evaluation—is not actually demonstrated in any reported experiment. The environment may well support it; the evidence doesn't show it. Second, the predicate weights and rubrics are author-defined. The equal-weight sensitivity analysis (Table 20) shows model ordering is stable, which is good, but it doesn't validate the predicates themselves, and the catalog lacks an independent audit. Third, there are no repeated-seed error bars on the model means; with one run per task per model, differences of a few points among the top models are not meaningful.\n\nThe citation pattern looks fair; the related work is accurately described and the crosswalk to ACP/UCP/AP2 is useful.\n\nWho gets value: anyone building or using agent evaluation environments for commerce or negotiation, especially the scoring trace idea and the verification protocol. It deserves a serious referee. I'd send it to review, and ask the authors to either demonstrate true two-sided policy interaction or rescope the contribution to what is actually measured.","headline":"Solid benchmark engineering with real validation, but the central claim of evaluating two independent policies together is not supported by the reported experiments.","tokens_in":27066,"tokens_out":1910,"would_cite":true,"duration_ms":20391,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ACWorld claims that agentic commerce can be evaluated reliably only when every agent action is validated before it changes shared market state, with full traces kept so that process evidence — not just final outcomes — decides the score.","keywords":["agentic commerce","vibe commerce","LLM agent evaluation","multi-agent environment","process-level verification","auditable traces","commerce benchmark","Buyer and Merchant policies"],"falsifier":"Build a second scorer for the same traces using independently authored predicates plus human adjudication of the 99 colliding runs; if the independent rubric finds the process signals add little, or disagrees with the paper's predicate verdicts on a large share of collisions, then the necessity of process-level evidence would be an artifact of the paper's own rubric rather than a property of the environment. A cheaper check: rerun the collision analysis with equal predicate weights and with predicates reordered, and confirm the 99/861 count and the model ordering persist.","tokens_in":26177,"feed_emoji":"🧾","tokens_out":6264,"duration_ms":64022,"temperature":0.7,"pith_summary":"ACWorld is built to answer a question existing benchmarks avoid: how should one evaluate Buyer and Merchant agents that share one market, each holding private goals and authority? The paper's answer is to separate deciding from committing — a Vibe Commerce Protocol binds every proposed action to its actor, a platform validates it, and only the world state may change, while the full decision-validation-effect record is kept for every step. The paper then argues, with evidence, that this record is not a luxury: across ten models, 99 of 861 imperfect runs reach the same final state as a fully correct run, so any evaluation that looks only at outcomes misses real errors. The benchmark reports model means of 66–86% on 200 capability tasks and 56–91% on 60 tasks searching a 785,022-listing catalog. If the argument holds, agent evaluation shifts from 'did the job finish' to 'can every step be verified.'","feed_headline":"99 failed commerce runs look identical to perfect ones","feed_subtitle":"An auditable agent-commerce benchmark shows why evaluations must verify process, not just outcomes.","key_machinery":"The load-bearing object is the Vibe Commerce Protocol (VCP), a contract layer that binds each action proposal to an authenticated actor and forbids messages from changing world state directly. Its companion is the linked record Lt = (actor, decision, validation, execution evidence, state delta) stored for every evaluated step, plus the deterministic transition W_{j+1} = F(W_j, u_j) through which the World applies only approved events. These records make a run reconstructable from the initial state and rescorable without another model call, which is what turns the paper's scores into verifiable claims rather than model self-reports. The benchmark scoring combines r declared Boolean or fractio","core_discovery":"The paper's central claim is that an agentic commerce environment can evaluate two independent, privately motivated policies in one persistent market if it separates agent decisions from the authority to change shared state. VCP attaches each proposed action to a named Buyer or Merchant, the Commerce Intelligence Platform accepts or rejects it against market rules, and only the World applies approved events through a deterministic transition; every step leaves a linked record of actor, decision, validation, evidence, and state change. The paper's empirical companion claim is that this process-level evidence is necessary: among 861 capability-coverage runs without full credit, 99 share a fina","pith_inferences":["The validate-then-commit separation is a general recipe for multi-agent evaluation beyond commerce: any shared-state domain with private actor objectives could adopt the same trace format, and the collision analysis is a template for measuring how much information a restricted evaluator view discards.","The 99/861 collision count is likely a lower bound on the blind spot of real deployed evaluators, because it is computed against this benchmark's own predicate verdicts; an independent label set for the same traces would show whether the claim survives outside the author-defined rubric.","Because incomplete trajectories retain positive signal on 1.88 of 3.49 predicates on average, a cheap testable extension is to use partial traces directly as training rewards before the full task is solved — the paper explicitly leaves the learning value of these signals unevaluated.","The headline scores should not be read as a general leaderboard: the paper frames them as characterizing these tracks, and the design invites re-weightings, new families, and adversarial tasks that would shift the ordering."],"forward_implications":["Evaluators that score only final outcomes will systematically undercount agent errors; the paper's collision numbers (99/861 for final state, 54/861 for final action, 45/861 for action sequence) quantify that blind spot on this benchmark.","Verified intermediate signals — reads, decisions, citations, actions — are candidate process rewards for training commerce agents, since traces rescore deterministically without another model call.","Separating decision from authority gives a reproducible protocol boundary: any extension (new product domain, tool, or world event) must pass the same validation and leave enough evidence to reconstruct its effects.","Because public tasks invite direct optimization, the paper commits to pairing future public development tasks with unpublished held-out test tasks to keep scores comparable.","Capability breadth and catalog-scale execution are distinct dimensions: model ordering differs between the two tracks, and large-catalog bottlenecks concentrate in Search and Decision rather than Validation."],"fun_headline_variants":["Final state hides 99 failed agent deals","Agent commerce audits need process, not just results","Why vibe commerce checks each step, not end state","One market, two private agents, zero outcome trust","Process beats results in agent commerce audits"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the benchmark's author-defined scoring predicates correctly capture what counts as a good grounding, negotiation, or purchase decision; if a predicate mis-specifies commercial intent, the reported scores and the 'final state alone misses errors' conclusion inherit that error — a risk the paper itself acknowledges by noting the catalogs have had no independent audit and that reconstruction verifies consistency, not semantic correctness.","fun_headline_variants_meta":{"raw":{"variants":["Final state hides 99 failed agent deals","Agent commerce audits need process, not just results","Why vibe commerce checks each step, not end state","One market, two private agents, zero outcome trust","Process beats results in agent commerce audits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000137,"raw_usage":{"total_tokens":965,"prompt_tokens":703,"completion_tokens":262,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":447,"completion_tokens_details":{"reasoning_tokens":192}},"tokens_in":447,"tokens_out":262,"duration_ms":3863,"temperature":1.0,"reasoning_tokens":192,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T07:14:36.500300+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a second scorer for the same traces using independently authored predicates plus human adjudication of the 99 colliding runs; if the independent rubric finds the process signals add little, or disagrees with the paper's predicate verdicts on a large share of collisions, then the necessity of process-level evidence would be an artifact of the paper's own rubric rather than a property of the environment. A cheaper check: rerun the collision analysis with equal predicate weights and with predicates reordered, and confirm the 99/861 count and the model ordering persist.","supporting_citations":[],"review_version":1}