{"id":"6fee2ecd-b651-418b-83eb-3ca8b43615ef","arxiv_id":"2605.16116","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ShopGym introduces ShopArena to convert live storefronts into self-contained sandbox shops and ShopGuru to synthesize 224 benchmark tasks, with validation showing structural preservation and positive correlation of agent performance between synthetic and live shops.","lead":"ShopGym turns live e-commerce sites into controllable, resettable simulations and generates grounded benchmark tasks across seven skill areas. This lets researchers test web agents in realistic yet repeatable settings instead of unstable live stores or narrow hand-built tests.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Correlation claim rests on only three synthetic + three real-data shops without reported coefficient or significance test","rationale":"The reader's weakest assumption directly flags potential systematic biases in the anonymized generation pipeline; the small shop count is the most immediate empirical threat to detecting or ruling out those biases. This is an internal methodological gap rather than an external-consensus disagreement, so the adjustment stays within CONDITIONAL rather than moving to REJECT.","tokens_in":1798,"tokens_out":341,"duration_ms":40354,"concrete_test":"Extract the per-shop average agent success rates (or equivalent primary metric) for the six shops from the results section or supplementary tables; compute Pearson (or Spearman) correlation between the three synthetic and three real-data shops and report the coefficient together with a two-sided p-value. If |r| < 0.5 or p > 0.05, the positive-correlation claim does not survive basic statistical scrutiny.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline result—that synthetic shops preserve structural properties and yield positively correlated agent performance—depends on behavioral evaluation across just six shops total. Graph-based structural metrics can be computed on any generated graph, but the performance correlation is the key external validity link to live storefronts. With N=6 (and likely fewer independent tasks per shop after aggregation), even a moderate observed correlation could arise from shop-specific artifacts in the staged generation process rather than faithful capture of navigation, catalog, and policy affordances. The abstract states the correlation exists but supplies neither the numerical value, the exact metric (success rate, steps, reward), nor controls for task difficulty or shop size.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces ShopGym, a framework for realistic simulation and scalable benchmarking of e-commerce web agents. Its core components are ShopArena, which converts live seed storefronts into self-contained sandbox shops via anonymized specifications and a staged, validated generation process, and ShopGuru, which synthesizes benchmark tasks across seven skill categories grounded in each shop's catalog, navigation structure, policies, and interaction affordances. The authors validate the approach through graph-based structural analysis and behavioral evaluation on 224 tasks across six sandbox shops (three synthetic, three real), claiming that the synthetic shops preserve key structural properties of live storefronts and that agent performance on synthetic shops is positively correlated with performance on live storefronts.","tokens_in":1923,"tokens_out":463,"duration_ms":63921,"significance":"If the central claims hold, ShopGym would address a key methodological bottleneck in e-commerce agent research by providing environments that are simultaneously realistic (grounded in live data), controllable, inspectable, and reproducible. The dual validation strategy combining structural graph metrics with behavioral agent runs on tasks derived from real storefront properties is a strength that could support more standardized and scalable evaluation protocols.","major_comments":[{"comment":"The headline claim that synthetic shops preserve structural properties and yield positively correlated agent performance (abstract) rests on behavioral evaluation across only six shops total. No correlation coefficient, p-value, exact performance metric (e.g., success rate or steps), or controls for task difficulty/shop size are reported, which is load-bearing for the external-validity argument that ShopArena faithfully captures navigation, catalog, and policy affordances without systematic biases.","section":null}],"minor_comments":[{"comment":"The abstract states the correlation exists but supplies limited detail on exact metrics, exclusion criteria, or statistical controls; adding these would improve clarity without altering the core contribution.","section":null},{"comment":"Clarify how the 224 tasks are distributed across the six shops and whether aggregation was performed before computing correlations.","section":null}],"recommendation":"major_revision","confidential_remarks":"The small N for the key correlation result may warrant requesting additional shops or statistical details in revision; the work appears a reasonable fit for an AI or agents-focused journal given its emphasis on simulation and benchmarking."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive review and for recognizing the potential of ShopGym to address methodological challenges in e-commerce agent evaluation. We address the major comment below, providing additional context from our experiments while committing to strengthen the reporting in revision.","responses":[{"response":"We agree that quantitative details on the correlation strengthen the external-validity argument and will add them in revision. The behavioral evaluation uses task success rate (binary completion of the specified shopping goal within a step budget) as the primary metric, averaged across the 224 tasks. We will report the Pearson correlation coefficient and p-value computed over the three matched shop pairs (synthetic vs. live), along with per-shop success rates and standard deviations. Task difficulty was controlled by generating tasks from the same seven skill categories with equivalent grounding in catalog size, navigation depth, and policy complexity for each pair; shop size was matched by selecting live and synthetic instances with comparable numbers of products and categories. The structural analysis (detailed in Section 4.1) uses graph metrics including average shortest path length, degree distribution, and clustering coefficient to demonstrate preservation independent of the behavioral results. While the sample of three pairs limits statistical power, the consistent positive trend across pairs supports the claim as preliminary evidence; we will explicitly note the small n as a limitation and outline plans for larger-scale validation.","revision_made":"yes","referee_comment":"The headline claim that synthetic shops preserve structural properties and yield positively correlated agent performance (abstract) rests on behavioral evaluation across only six shops total. No correlation coefficient, p-value, exact performance metric (e.g., success rate or steps), or controls for task difficulty/shop size are reported, which is load-bearing for the external-validity argument that ShopArena faithfully captures navigation, catalog, and policy affordances without systematic biases."}],"tokens_in":1446,"tokens_out":390,"duration_ms":39864,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing here is that ShopGym converts live storefront seeds into self-contained sandbox shops via anonymized specs and a staged generation process, then layers on grounded task synthesis across skill categories. This directly tackles the live-versus-sandbox tradeoff that has limited agent evaluation in e-commerce. The framework ships two named pieces—ShopArena for the simulation layer and ShopGuru for task creation—and they validate it with graph-based structural checks plus agent runs on 224 tasks spread over six shops, three built from synthetic data and three from real data. The claim is that the synthetic versions keep enough of the navigation, catalog, and policy structure that agent performance correlates positively with live storefronts. That is the concrete advance: a reproducible pipeline that starts from real data rather than hand-crafted toys. It is useful because it produces resettable, inspectable environments that still aim to reflect actual shopping affordances. The approach is grounded in external live seeds, so there is no obvious circularity or free parameters driving the results. The validation protocol itself is a reasonable first step for this kind of work. The soft spot is the behavioral correlation. It rests on only six shops total, and the abstract supplies neither the actual coefficient, the precise success metric, nor any controls for shop size or task difficulty. With so few independent cases, even a moderate positive link could reflect artifacts in how the staged generation was tuned rather than faithful capture of the properties that matter for agents. If the full paper adds more shops, reports the numbers, or shows robustness checks, the result strengthens; otherwise the external-validity link stays preliminary. This paper is aimed at groups building web agents for retail or similar interactive domains who need scalable, reproducible testbeds. Readers who want a practical framework with some empirical backing will find it worth reading. It deserves a serious referee because the problem is real, the components are clearly described, and the validation direction is honest even if the current evidence is thin. I would send it for review and ask specifically for the correlation details and any additional shops or controls.","headline":"ShopGym gives a workable path from live e-commerce sites to controllable, inspectable benchmarks, but the key correlation result sits on a very small sample of shops with no numbers or controls shown.","tokens_in":2413,"tokens_out":493,"would_cite":false,"duration_ms":73578,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"E-commerce web-agent simulation framework has no structural overlap with RS forcing chain","alignment":"orthogonal","rationale":"The paper's core machinery (ShopArena's anonymized specification synthesis + staged code generation, ShopGuru's grounded task generators, graph-based structural metrics, and behavioral correlation validation) operates entirely in the domain of scalable AI benchmarking environments. It contains no recognition-cost functions, golden-ratio identities, 8-tick periodicity, J-cost forcing, or derivations of physical constants from a single distinction. RS theorems such as reality_from_one_distinction and the J-cost uniqueness results in Cost/FunctionalEquation.lean are therefore neither confirmed nor contradicted.","tokens_in":56858,"confidence":"high","tokens_out":154,"duration_ms":17151,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"ShopGym turns live e-commerce sites into controllable sandbox shops that keep the same structural properties and produce matching agent performance signals.","keywords":["e-commerce web agents","simulation environments","benchmark tasks","synthetic shops","agent evaluation","ShopArena","reproducible benchmarks"],"falsifier":"A controlled experiment in which the same agents are run on both the generated sandbox shops and their corresponding live storefronts, showing that success rates, error patterns, or performance rankings fail to correlate.","tokens_in":2703,"feed_emoji":"🛍️","tokens_out":712,"duration_ms":37263,"temperature":0.7,"pith_summary":"The paper presents ShopGym as a framework to build simulation environments and benchmark tasks for e-commerce web agents. Its ShopArena component converts real storefronts into self-contained sandbox versions using anonymized specifications and a staged generation process. ShopGuru then creates tasks grounded in each shop's catalog, navigation, policies, and interaction patterns. This setup aims to combine the realism of live sites with the control, reproducibility, and scalability that hand-built tests lack. Validation on six shops and 224 tasks shows that synthetic versions preserve key structures and that agent results on them correlate positively with results on the original live sites.","feed_headline":"Synthetic shops match live e-commerce agent performance","feed_subtitle":"Framework converts real storefronts into controllable sandboxes whose structural properties and task outcomes align with the originals.","key_machinery":"ShopArena, the simulation layer that converts live seed storefronts into self-contained sandbox shops through anonymized shop specifications and a staged, validated generation process.","core_discovery":"ShopGym produces self-contained, resettable, inspectable, and stable evaluation artifacts that preserve structural properties and agent-evaluation signals relevant to shopping tasks. ShopArena converts live seed storefronts into sandbox shops through anonymized shop specifications and a staged, validated generation process. On top of these, ShopGuru synthesizes benchmark tasks across seven skill categories while grounding each task in the shop's catalog, navigation structure, policies, and interaction affordances. Results from graph-based structural analysis and agent-based behavioral evaluation confirm that the synthetic shops maintain key properties of live storefronts and that agent性能 on ","pith_inferences":["The approach could support iterative agent development by allowing rapid resets and targeted variations without hitting live-site rate limits or non-stationarity.","If the correlation between synthetic and live performance holds for a wider range of agent architectures, the framework might serve as a pre-deployment filter before live testing.","Similar staged-generation methods could be applied to other interactive web domains such as travel booking or news reading to create analogous controlled environments."],"forward_implications":["Evaluation of web agents can use many diverse shops while remaining fully reproducible and inspectable.","Benchmark tasks stay grounded in real catalog, navigation, and policy details rather than abstract templates.","Agent performance measured in the synthetic environments tracks performance on the live versions they derive from.","The same seed storefronts can generate multiple controlled variants for systematic comparison across skill categories."],"fun_headline_variants":["ShopGym turns real stores into stable e-commerce sandboxes","Synthetic shops retain live e-commerce properties for agent tests","Framework creates reproducible benchmarks preserving shop structures","Agent performance correlates on synthetic and real e-commerce shops"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The anonymized shop specifications and staged generation process in ShopArena capture the essential navigation structure, catalog, policies, and interaction affordances of original live storefronts without introducing systematic biases that alter agent behavior or evaluation signals.","fun_headline_variants_meta":{"raw":{"variants":["ShopGym turns real stores into stable e-commerce sandboxes","Synthetic shops retain live e-commerce properties for agent tests","Framework creates reproducible benchmarks preserving shop structures","Agent performance correlates on synthetic and real e-commerce shops"]},"model":"grok-4.3","cost_usd":0.008084,"raw_usage":{"total_tokens":3652,"prompt_tokens":783,"num_sources_used":0,"completion_tokens":59,"cost_in_usd_ticks":80840500,"prompt_tokens_details":{"text_tokens":783,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2810,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":783,"tokens_out":59,"duration_ms":53409,"temperature":1.0,"reasoning_tokens":2810,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-20T17:51:27.232975+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A controlled experiment in which the same agents are run on both the generated sandbox shops and their corresponding live storefronts, showing that success rates, error patterns, or performance rankings fail to correlate.","supporting_citations":[],"review_version":1}