{"id":"f584031d-3a39-4594-881d-f1683e39e901","arxiv_id":"2508.04266","paper_version":4,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"ShoppingBench is an e-commerce agent benchmark with realistic multi-step intents; GPT-4.1 succeeds under 50% of the time, and a distilled smaller model approaches its performance.","lead":"This paper presents ShoppingBench, a new benchmark that tests AI shopping agents on complex real-world tasks such as applying vouchers and managing budgets. It also describes a method to distill a large language agent's shopping abilities into a smaller model that performs nearly as well.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Under-50% success claim rests on an unvalidated automatic success metric; no human agreement or task-difficulty calibration is reported in the abstract.","rationale":"The reader's weakest_assumption targets the faithfulness of generated instructions and the sandbox as a proxy for real-world shopping, which is a broad construct-validity concern. My concern is more specific and logically prior: even within the sandbox, the success metric could be mismeasuring agent performance. This falls under the same general assumption but is a concrete, testable flaw that could invalidate the specific under-50% number while leaving the benchmark concept intact. Since the review is abstract-only, I cannot confirm the scorer is flawed; I only note that the abstract provides no evidence against this possibility. The reader's UNVERDICTED status with LOW confidence remains appropriate, so I recommend no change to the verdict. My agreement is partial because the reader's assumption is broader than my pointed concern; both are relevant, but mine is the more load-bearing for the central claim as stated.","tokens_in":601,"tokens_out":1998,"duration_ms":23756,"concrete_test":"Sample 200 tasks from ShoppingBench and run GPT-4.1 (and ideally the distilled agent) to obtain full trajectories. Have two independent human annotators, blind to the automatic score, label each trajectory as success or failure, with clear rubrics for partially correct solutions. Compute (a) inter-annotator agreement (Cohen's kappa) and (b) the human-judged success rate versus the automatic success rate. If kappa < 0.6, or if the automatic rate deviates from the human rate by more than 10 percentage points in either direction, the under-50% claim is not a reliable measure of agent capability and must be re-reported with a corrected metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that GPT-4.1 achieves under 50% absolute success depends entirely on how 'success' is scored in the simulated sandbox. The abstract provides no details on the scoring mechanism: is it a rule-based state check, a string match against a single ground-truth answer, or an LLM judge? If the scorer is strict or ambiguous, a high false-negative rate could depress all agent scores uniformly, making the under-50% result an artifact of evaluation harshness rather than evidence of task difficulty. In particular, tasks such as 'find multi-products seller' appear to have multiple valid solutions; a single-answer matcher would penalize legitimate alternatives. Similarly, if the automatically generated instructions are under-specified or self-contradictory, agents might fail through no fault of their reasoning ability. The abstract also does not report human performance on the same tasks or any calibration check. Absent human validation of the generated tasks and of agent trajectories, the strong claim that SOTA agents are below 50% is not yet supported. This is not an argument that the benchmark is wrong in principle, but that the measurement instrument itself needs validation before the headline number can be interpreted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ShoppingBench introduces a new benchmark for evaluating LLM-based agents on complex e-commerce intents (e.g., applying vouchers, budget management, finding multi-product sellers). It uses a scalable instruction-generation framework over sampled real-world products and a large simulated shopping sandbox containing over 2.5 million products. The paper reports that state-of-the-art agents such as GPT-4.1 achieve absolute success rates under 50% on these tasks. It also proposes a trajectory-distillation method combining supervised fine-tuning and reinforcement learning to train a smaller agent that is claimed to reach competitive performance with GPT-4.1.","tokens_in":911,"tokens_out":1988,"duration_ms":22856,"significance":"If the results hold, ShoppingBench would address a real gap in e-commerce benchmarks, which typically focus on simple finding/buying intents. The scale of the sandbox and the attempt to ground tasks in sampled real-world products are notable strengths. The under-50% result for strong agents would be a valuable calibration point for the community. However, because the submission under review is abstract-only, these claims cannot be independently assessed. The significance will depend on the validation of the evaluation protocol and the reproducibility of the benchmark.","major_comments":[{"comment":"The central claim that GPT-4.1 achieves under 50% success is stated without specifying the success metric, number of tasks per difficulty level, or any statistical uncertainty (e.g., confidence intervals or number of runs). The reader cannot determine whether success is a rule-based state check, a string match, or an LLM judge. For tasks such as 'find multi-products seller', multiple valid solutions may exist; a strict single-answer scorer could systematically lower all agents' scores, making the under-50% result an artifact of evaluation harshness. The abstract should at least state the scoring mechanism and, ideally, report human performance or inter-annotator agreement as a calibration check.","section":"Abstract"},{"comment":"The automatically generated instructions are claimed to simulate user intents, but no validation is reported. If the instructions are under-specified, contradictory, or unnatural, agent failures may reflect instruction noise rather than task difficulty. The manuscript should provide evidence of instruction quality, such as human plausibility ratings, consistency checks, or a comparison of generated tasks against human-authored tasks. Without such validation, the headline under-50% claim is not yet supported.","section":"Abstract"},{"comment":"The trajectory distillation result is described only as 'competitive performance compared to GPT-4.1.' This is too vague to evaluate. The abstract should report the actual success rates of the distilled agent, the GPT-4.1 teacher, and baselines, along with task-level breakdowns and any variance. The claim appears load-bearing for the second contribution, but as written it is not falsifiable.","section":"Abstract"}],"minor_comments":[{"comment":"The phrase 'find multi-products seller' is grammatically awkward and could be clarified (e.g., 'find a seller offering multiple products').","section":"Abstract"},{"comment":"The term 'end-to-end shopping benchmark' is not defined; the reader can infer it means a full interactive agent evaluation, but a precise definition would help.","section":"Abstract"},{"comment":"The provenance of the '2.5 million real-world products' is not mentioned; providing a source and a description of any filtering/processing would strengthen reproducibility.","section":"Abstract"},{"comment":"No reference to prior e-commerce benchmarks is provided in the abstract; placing the work in context would help the reader judge novelty.","section":"Abstract"}],"recommendation":"uncertain","confidential_remarks":"This review is based solely on the abstract because no full text was available. The major concerns center on the unvalidated success metric and the vague distillation claim. If the full paper provides a detailed evaluation protocol, human validation, and a quantitative comparison, the concerns may be addressed; however, as presented, the evidence is insufficient to render a confident verdict. I recommend evaluating the full manuscript before making a final decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a plausible and useful benchmark for complex shopping intents, and the distillation bit is a bonus. The abstract claims GPT-4.1 under 50% on their tasks — that's the kind of number people will quote, so it matters that the evaluation protocol be airtight. Right now we only have the abstract, so we can't see the scoring mechanism, task counts, or human baselines. The stress-test worry is fair: if their success checker is a single-answer matcher, tasks like 'find multi-products seller' would have a hidden false-negative rate, and the under-50% result could be an artifact of strict scoring rather than a statement about agent capability. They don't report human performance or calibration in the abstract, so we can't yet separate task difficulty from evaluation harshness. That said, this is not a fatal flaw — it's an unresolved detail. The benchmark itself is new in scope: vouchers, budget constraints, multi-product sellers, plus a 2.5M-product sandbox grounded in real-world data. That's a real step beyond the find/buy benchmarks. The distillation part (SFT plus RL on synthetic trajectories) is standard machinery, but applying it to this task domain is reasonable, and 'competitive with GPT-4.1' is vague but not wrong. I'd want to see the full paper before trusting the numbers, but I'd definitely send it to review. A good referee would ask for: the exact success metric, human agreement or at least a small hand-validated set, task-level breakdowns, and hyperparameters for the distillation. If those check out, this could become a standard benchmark. If not, the headline will be misleading. My recommendation: engage with the full paper; keep the abstract claim in the 'promising but unverified' bucket until the evaluation details are public.","headline":"Benchmark fills a real gap in e-commerce agent evaluation, but the headline under-50% number is only as good as the scoring and calibration, which the abstract doesn't show.","tokens_in":1305,"tokens_out":1621,"would_cite":true,"duration_ms":17456,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new benchmark built on 2.5 million real products reports that GPT-4.1 and similar state-of-the-art agents complete fewer than half of its realistic shopping tasks.","keywords":["shopping agents","LLM benchmark","intent grounding","e-commerce","trajectory distillation","reinforcement learning","sandbox simulation","instruction generation"],"falsifier":"Collect a sample of real online shopping sessions from user logs, convert them into ShoppingBench-style tasks, and compare agent success rates on those tasks against success on the generated ones. If real-session tasks are systematically easier or harder, the under-50% claim would not generalize. Alternatively, a human evaluation showing that generated instructions contain inconsistent constraints (e.g., vouchers that cannot apply to the sampled products) would undermine the benchmark's validity.","tokens_in":578,"feed_emoji":"🛒","tokens_out":3571,"duration_ms":38091,"temperature":0.7,"pith_summary":"ShoppingBench tests language-model agents on realistic e-commerce goals, not just product search and purchase. It generates user instructions that bundle the kinds of constraints real shoppers juggle, such as applying vouchers, respecting budgets, and coordinating multiple products from one seller. The authors report that state-of-the-art agents like GPT-4.1 succeed on fewer than half of the tasks, which they take as evidence that current agents lack robust intent grounding. The paper also shows that a smaller agent trained on distilled trajectories from a larger model can come close to GPT-4.1's performance.","feed_headline":"Top language agents score under 50% on real-world shopping tests","feed_subtitle":"A sandbox with 2.5 million real products shows LLM agents struggle with budgets, vouchers, and multi-product goals.","key_machinery":"The intent-grounded instruction generator is the load-bearing component: it samples real-world products and composes user instructions that combine multiple grounded intents, which are then executed in the shopping sandbox. The sandbox, with 2.5 million real products, provides consistent, automated verification of task success. For the distillation result, synthetic trajectories from a strong agent are used to supervise a smaller agent through supervised fine-tuning and reinforcement learning.","core_discovery":"The central claim is that existing e-commerce benchmarks underestimate what real shopping requires, and that the gap can be measured with a scalable benchmark of intent-grounded tasks. ShoppingBench builds instructions by drawing on real-world product data and layering intents such as voucher use, budget management, and multi-product seller constraints. In an interactive sandbox containing over 2.5 million real products, even the strongest evaluated agents—represented by GPT-4.1—achieve absolute success rates below 50%. The paper further claims that the capabilities of a large teacher agent can be distilled via supervised fine-tuning and reinforcement learning on synthetic trajectories, yiel","pith_inferences":["A natural extension the paper leaves implicit is testing the same instruction-generation recipe outside shopping, such as travel or office workflows, to see whether intent-grounded complexity reveals similar agent gaps.","The benchmark's generated instructions could be compared against logged real user sessions to quantify any distribution shift; if real intents differ systematically, the under-50% number would need recalibration.","One testable implication is that asking a human panel to attempt the same tasks would place a human ceiling, making the agent shortfall more interpretable.","Because the distillation uses synthetic trajectories, it remains an open question whether the student agent inherits the teacher's failures on novel task combinations."],"forward_implications":["If ShoppingBench is a fair proxy, any agent meant for practical e-commerce use will need to handle combined constraints, not just item lookup, before it can be trusted.","The under-50% ceiling for GPT-4.1 provides a concrete headroom figure that future agent designs can be measured against.","The report of a distilled small agent competitive with GPT-4.1 implies that shopping-agent capabilities may be compressible for deployment at lower cost.","The sandbox allows reproducible comparison across agents, which the field lacks for multi-intent shopping."],"supporting_citations":[],"fun_headline_variants":["LLM agents score under 50% on realistic shopping tasks","Real-world shopping benchmark stumps GPT-4.1","2.5M-product sandbox exposes AI shopping shortfalls","Vouchers, budgets, multi-product goals trip up LLM agents","Even GPT-4.1 struggles on realistic shopping benchmark"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The automatically generated instructions and the simulated shopping environment are a faithful, unbiased proxy for real-world user intents and purchasing constraints.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents score under 50% on realistic shopping tasks","Real-world shopping benchmark stumps GPT-4.1","2.5M-product sandbox exposes AI shopping shortfalls","Vouchers, budgets, multi-product goals trip up LLM agents","Even GPT-4.1 struggles on realistic shopping benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000913,"raw_usage":{"total_tokens":3749,"prompt_tokens":723,"completion_tokens":3026,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":2940}},"tokens_in":467,"tokens_out":3026,"duration_ms":22805,"temperature":1.0,"reasoning_tokens":2940,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:42:48.484361+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a sample of real online shopping sessions from user logs, convert them into ShoppingBench-style tasks, and compare agent success rates on those tasks against success on the generated ones. If real-session tasks are systematically easier or harder, the under-50% claim would not generalize. Alternatively, a human evaluation showing that generated instructions contain inconsistent constraints (e.g., vouchers that cannot apply to the sampled products) would undermine the benchmark's validity.","supporting_citations":[],"review_version":1}