{"paper":{"title":"E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation","arxiv_id":"2608.30730","primary_cat":"cs.LG","submitted_at":"2026-08-31T13:03:57+00:00","version_compiled":"v1"},"notice":"Every quoted sentence is the authors' text from the arXiv listing abstract. Labels and probabilities are the outputs of classifier-1.13.0 on the question bank abstract-p1-20260923, exactly as returned. No sentence in this record was written by a language model.","schema":"pith.compiled-record/v0","status":"compiled","abstract":{"source":"arXiv listing abstract v1","sentences":[{"i":0,"p":0.99,"id":"s0","own":false,"role":"background","text":"Long-horizon agentic tasks go beyond chaining short tasks over more interaction turns.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.07,"main_claim_p":0.03,"quantitative":false,"quantitative_p":0.25,"role_probabilities":{"gap":0.01,"method":0.0,"result":0.0,"background":0.99,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":1,"p":0.99,"id":"s1","own":false,"role":"background","text":"Their evolving dynamic environments and long-range dependencies require Large Language Models (LLMs) to continually explore, learn from experience, and adapt their policies over thousands of steps.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.06,"main_claim_p":0.03,"quantitative":false,"quantitative_p":0.43,"role_probabilities":{"gap":0.01,"method":0.0,"result":0.0,"background":0.99,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":2,"p":1.0,"id":"s2","own":true,"role":"contribution","text":"We introduce E-Commerce Bench, the first open-source benchmark that integrates multi-round counterpart negotiation and dynamic events into a year-long business operation.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.95,"main_claim_p":0.07,"quantitative":false,"quantitative_p":0.32,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":1.0,"interpretation":0.0}},{"i":3,"p":0.58,"id":"s3","own":true,"role":"method","text":"Over a 365-day year, an LLM agent concurrently runs multiple online stores, researching the market, negotiating with suppliers to source inventory, optimizing sales strategies, fulfilling orders, handling returns, and managing cash flow to maximize its end-of-year total assets.","hedge":{"p":0.99,"label":"asserted"},"own_p":0.63,"main_claim_p":0.05,"quantitative":true,"quantitative_p":0.93,"role_probabilities":{"gap":0.0,"method":0.58,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.42,"interpretation":0.0}},{"i":4,"p":0.99,"id":"s4","own":true,"role":"method","text":"To construct a realistic merchant-side operating environment, the product and supplier data are derived from a real e-commerce platform, while a year-long calendar of promotions, natural disasters, and supply-chain shocks continually reshapes demand.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.88,"main_claim_p":0.02,"quantitative":false,"quantitative_p":0.36,"role_probabilities":{"gap":0.0,"method":0.99,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.01,"interpretation":0.0}},{"i":5,"p":1.0,"id":"s5","own":true,"role":"method","text":"For reproducibility, both sides of the market are deterministic: customer purchases and returns follow a fixed demand model, while a negotiation kernel determines supplier pricing, concessions, and decisions, with an LLM used only to verbalize them.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.84,"main_claim_p":0.03,"quantitative":false,"quantitative_p":0.07,"role_probabilities":{"gap":0.0,"method":1.0,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":6,"p":0.95,"id":"s6","own":true,"role":"result","text":"We evaluate 18 frontier models across seven dimensions, including year-end assets, and find that no single model dominates.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.97,"main_claim_p":0.48,"quantitative":true,"quantitative_p":0.93,"role_probabilities":{"gap":0.0,"method":0.03,"result":0.95,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.02,"interpretation":0.0}},{"i":7,"p":1.0,"id":"s7","own":true,"role":"result","text":"GPT-5.6 Sol earns the most, growing the 100,000 opening stake into 1,431,425, yet it ranks 16th of 18 on fraud avoidance and trails Fable5 in operational efficiency.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.94,"main_claim_p":0.14,"quantitative":true,"quantitative_p":0.99,"role_probabilities":{"gap":0.0,"method":0.0,"result":1.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":8,"p":1.0,"id":"s8","own":true,"role":"result","text":"Among open-weight models, Qwen3.8-Max-Preview leads with 416,252, 38% above GLM 5.2 (high), and achieves the strongest learning over the horizon, progressively bargaining down prices across repeated orders.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.95,"main_claim_p":0.11,"quantitative":true,"quantitative_p":0.99,"role_probabilities":{"gap":0.0,"method":0.0,"result":1.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":9,"p":0.84,"id":"s9","own":true,"role":"contribution","text":"Our code is available at https://github.com/QwenLM/E-CommerceBench.","hedge":{"p":0.97,"label":"asserted"},"own_p":0.69,"main_claim_p":0.02,"quantitative":false,"quantitative_p":0.03,"role_probabilities":{"gap":0.0,"method":0.06,"result":0.0,"background":0.01,"limitation":0.0,"future_work":0.09,"contribution":0.84,"interpretation":0.0}}]},"compiled":{"main_claim":{"p":1.0,"text":"We evaluate 18 frontier models across seven dimensions, including year-end assets, and find that no single model dominates.","location":"abstract s6","hedge_strength":{"p":1.0,"label":"asserted"}},"provenance":{"route":{"via":"TypeSafe API","model":"classifier-1.13.0","endpoint":"https://api.typesafe.ai/v1/systemone","provider":"TypeSafe"},"packing":"one request per paper","requests":1,"input_tokens":7813,"cost_usd_reported":0.0},"compiled_at":"2026-09-23","bank_version":"abstract-p1-20260923","model_version":"classifier-1.13.0","abstract_states":{"claims_first":0.95,"states_limitation":0.52,"human_participants":0.03,"controlled_comparison":0.12,"randomized_assignment":0.02,"states_quantitative_result":0.99,"compares_with_prior_results":0.6,"mentions_code_or_data_availability":0.98},"evidential_basis":{"p":0.53,"value":"observation or experiment","probabilities":{"argument":0.0,"literature":0.0,"proof_or_derivation":0.0,"analysis_of_existing_data":0.0,"observation_or_experiment":0.53,"simulation_or_computation":0.47}},"contribution_type":{"p":1.0,"value":"dataset or benchmark","probabilities":{"review":0.0,"method_or_tool":0.0,"empirical_study":0.0,"simulation_study":0.0,"theoretical_result":0.0,"dataset_or_benchmark":1.0,"position_or_commentary":0.0}},"contribution_statement":{"p":1.0,"text":"We introduce E-Commerce Bench, the first open-source benchmark that integrates multi-round counterpart negotiation and dynamic events into a year-long business operation.","location":"abstract s2"}},"coverage":{"tier":"abstract","levels_run":["P abstract"],"not_examined":["full text","figures and tables","references and citing sentences","relations to other papers"],"sources_read":["arXiv listing abstract v1"],"levels_not_run":["S sections","T body sentences","C citations","R cross-paper relations"],"sentences_examined":10,"sentences_in_abstract":10},"measured_accuracy":{"url":"https://pith.science/accuracy","measured_on":"2026-10-02","method":"two independent model families labelled a fresh random sample blind; the figure is how often the published label matches their consensus","sentence_role_agreement":0.92,"main_claim_agreement":0.908,"typed_citation_agreement":0.933,"typed_citation_agreement_95_interval":[0.917,0.945],"review_comments_accurate":0.758,"review_comments_accurate_95_interval":[0.684,0.82],"review_caught_admitted_error_found_or_related":0.448,"review_comments_judged":{"judged_by":["gpt-5.6-luna","grok-4.7"],"judged_against":"Pith's own text copy of each paper: text extracted from the PDF, the first 90,000 characters","measured_on":"2026-10-04","accurate":113,"decided":149,"sampled_reviews":200,"split_by_lane":false,"by_lane":"https://pith.science/accuracy#by-lane"}}}