Artificial Intelligence · arXiv:2610.36980 v2 · submitted 27 Oct 2026
AutoVend: Benchmarking LLM Agents on Long-Horizon Autonomous Vending Operations
The paper says
“Large language model (LLM) agents are extending agent-mediated commerce on the Web from shopping for consumers to operating businesses for sellers. In online-to-offline (O2O) commerce, decisions are issued through Web-based platforms but realized through costly physical execution.”
more of the abstract
Evaluating such agents requires going beyond individual transactions to ask whether decisions made online turn into sustained economic returns. We introduce AutoVend, a benchmark in which one LLM agent manages 20 vending machines across five site types through a Web-based operating interface for 180 simulated days. Its data-grounded simulator combines a shared warehouse, procurement lead times, paid replenishment, inventory expiration, and recurring operating costs. We evaluate five LLM agents against a rule-based inventory-control baseline and diagnose their site coordination, execution, and performance over time. Four of five evaluated LLM agents fall short of this baseline, and returns per dollar of inference cost rank models differently from returns alone. Operational analyses show that gains in individual metrics do not consistently yield stronger economic growth, as inventory and execution costs erode these gains. Trajectory analysis further reveals difficulties in grounding feasible actions, managing inventory commitments, and converting feedback into effective policy revisions.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 28 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
The passes disagreed. Reader: holds up. Skeptic: holds up only with conditions. Referee: minor revision.
AutoVend is a well-designed benchmark for seller-side LLM agent evaluation, but its central claim rests on a single random seed and needs broader validation. This line is wrong
2 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Section 4.1 and Table 2
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.63):
“Future work should test these findings, drawn from three runs per model in one environment, across varied operating conditions and controlled interventions such as explicit state verification and persistent pol- icy updates.”
Pith says: Objection 1 of 2 · major
The central quantitative result that four of five LLM agents underperform the rule-based baseline is based on a single demand seed (all runs use 'one frozen world with the same random seed'). While the baseline is reported to be stable across seeds 1–3 (CV 1.1%), the LLM agents exhibit substantial variance (e.g., Kimi K3 CV 45.6%).
more of objection 1
Without testing over multiple demand realizations, the robustness of the ranking—and of the headline claim itself—is unknown. The paper acknowledges this as future work (sentence b222), but it is a load-bearing limitation for the paper's main conclusion. I recommend either (a) running experiments on at least 2–3 additional seeds with a representative subset of agents (e.g., the best and the worst) to assess rank stability, or (b) prominently qualifying the claim in the abstract and conclusion as preliminary and seed-dependent.
The paper says, at Section 3.3
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.91):
“Price and environmental responses remain unvalidated by design. 5 100 101 102 103 Simulated units 100 101 102 103 Realized units Worksite Campus Residential Commercial Office Unknown TeaWaterEnergySodaSportsJuiceMilkJerkyNoodlePastryCoffeeCandyCookieOther 0% 5% 10% 15% 20% 25% 30%Share of units sold Real Sim (a) Weekly sales per machine (b) Product mix Figure 3: Held-out evaluation of the demand model on 67 machines over an eight-week test window, averaged over 100 seeds.”
Pith says: Objection 2 of 2 · major
The paper states that price and environmental responses 'remain unvalidated by design' (sentence b110). Since agents make pricing decisions and respond to weather/holidays, the simulation's accuracy for these components is unknown. While the demand volume and product mix are validated, the unvalidated price elasticities and weather multipliers could disproportionately affect agents that rely on dynamic pricing or weather-aware strategies.
more of objection 2
I recommend adding a discussion of the potential direction of bias this introduces and, if feasible, a sensitivity analysis over plausible ranges for the price elasticities and environmental response parameters.
6 minor objections
The paper says, at Abstract
Pith says: minor objection 1 of 6
The phrase 'returns per dollar of inference cost rank models differently from returns alone' is interesting but could be misinterpreted. Clarify that the ranking by API efficiency (net-asset growth per USD of API cost) differs from the ranking by terminal net assets.
The paper says, at Table 2
Pith says: minor objection 2 of 6
The table reports 'Shelf Utilization' as a percentage (e.g., 88.91%), but the text later uses a different metric: the 'share of site-days with empty shelves' (e.g., 25.21% for Claude). Consider adding the empty-site-days fraction directly to the table or providing a cross-reference to make the two inventory-availability metrics easier to compare.
show 4 more
The paper says, at Figure 5(b)
Pith says: minor objection 3 of 6
The reassignment gap is defined as a percentage, but the denominator is not explicitly stated in the caption. Clarify how the gap is computed—what is the range and what does 0% correspond to? This would improve interpretability.
The paper says, at Section 4.1 (Baseline description)
Pith says: minor objection 4 of 6
The rule-based baseline is described as using newsvendor critical ratios and periodic order-up-to replenishment, with prices 'adjusted through local gross-profit trials.' Please provide a brief specification of the price-adjustment mechanism (e.g., how often, what step size) to allow reproducibility.
The paper says, at Section 5.2
Pith says: minor objection 5 of 6
The observation that 'all 15 LLM executions finish above the initial capital' is a notable positive result, but it is not mentioned in the conclusion. Consider highlighting it as evidence that even underperforming agents can generate returns, even if they fail to match the baseline.
The paper says, at Throughout
Pith says: minor objection 6 of 6
Minor: ensure consistency in model naming—'GPT-6.1 Sol' vs 'GPT-6.1 Sol' is consistent; double-check capitalization of model names (e.g., 'DeepSeek V4.1 Flash' appears correctly). No typos were found in the reviewed text.
What would settle it
Run every LLM agent (and the rule-based baseline) on each of at least 5 independent world seeds (e.g., seeds 1–5), using 3 LLM runs per seed. Compute the fraction of (seed, LLM-run) pairs where each LLM's terminal net assets exceed the baseline's mean across its own 3 runs per seed. If for any LLM that fraction exceeds 0.5 on average, the claim that it 'underperforms' is not robust across world seeds.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 28 Oct 2026
- Reader holds up
- Skeptic disagrees: says it holds up only with conditions Main result rests on a single world seed; generalizability of the 'four of five underperform' claim is untested.
- Editor wrote the verdict above
- Referee minor revision: 2 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
You should know two things about this paper. First, AutoVend is a solidly built benchmark for long-horizon vending operations: 20 heterogeneous machines, shared inventory, procurement lead times, paid site visits, and a demand model grounded in real sales records. The validation against held-out data is credible (WAPE 0.473, TVD 0.153), and the operational diagnostics (assortment, waste, pricing regret, capital efficiency) are thorough. The paper does what a good benchmark paper should: it builds a plausible environment, checks that it behaves like real data, and then runs agents. Second, the headline result—four of five LLM agents underperform a rule-based baseline—is interesting but not yet robust. The experiments use a single world seed, with only three runs per LLM to capture agent stochasticity. The authors acknowledge this in their conclusion, but it remains a load-bearing limitation: we don’t know whether the ranking holds across different demand paths. The paper also states that price and environmental responses are unvalidated by design; given that agents adjust prices and respond to weather, this could affect the realism of the evaluation. These are honest limitations, but they constrain what we can confidently say from the results. The benchmark itself is a real contribution—it fills a gap between single-store simulations and multi-site physical retail. The toolkit and evaluation protocol are reproducible. The paper deserves peer review; the editor should send it. My advice: accept it after the authors add experiments across multiple seeds (or provide a convincing argument that seed variation doesn’t change the qualitative result) and acknowledge the unvalidated components more prominently. This is a useful resource for anyone working on agent-mediated commerce or long-horizon planning.
The strongest objection
The central claim that four of five LLM agents underperform the rule-based baseline (sentence b219) is based on evaluations performed on a single frozen world seed (sentence "all in one frozen world with the same random seed"). Variation across LLM runs captures only agent stochasticity, not variation in the demand path. The baseline's stability across seeds 1–3 (CV 1.1%) does not guarantee that LLM performance relative to the baseline is similarly stable across different demand realizations. Given the high variance of Kimi K3 (CV 45.6%) even on one seed, and the moderate variance of others (~11%), different seeds could shift rankings. The paper acknowledges this only as future work (sentence b222: "drawn from three runs per model in one environment"), but it is a load-bearing limitation because the headline quantitative result is not demonstrated to be robust across the stochasticity that the paper intentionally built into the simulator (overdispersed shocks, auto-correlation, weather, holidays). A claim that a majority of agents underperform a baseline should be tested over multiple demand paths to establish its reliability.
The reader identified the unvalidated price/environmental responses as the weakest assumption, but that concern targets realism rather than the reliability of the central comparative result within the simulation framework. The more immediately load-bearing concern for the paper's main claim is that the comparison rests on a single random seed. The authors acknowledge the limitation but do not mitigate it in the present work. While the benchmark and diagnostics are still valuable, the headline result that 'four of five underperform' is not yet established as a robust finding. A conditional verdict—acceptance contingent on multi-seed validation—is appropriate.
The referee's report
AutoVend fills an important gap in seller-side agent evaluation by combining multi-site coordination, physical execution costs, and long-horizon economic returns. The simulator is grounded in real vending sales records and validated on demand volume and product mix. The finding that most evaluated LLM agents fail to beat a straightforward algorithmic baseline is striking and provides a clear, reproducible challenge for future work on long-horizon autonomous commerce agents. The benchmark is well-designed, the evaluation protocol is thorough, and the diagnostics offer insight beyond a single aggregate metric. The paper is likely to be influential in the agent-mediated commerce community.
Pith's reading of the paper (a machine's summary, not the authors')
The paper introduces AutoVend, a benchmark where a single LLM agent manages 20 vending machines across five site types for 180 simulated days. The environment includes shared inventory, procurement lead times, paid replenishment, and product expiration, reflecting the physical costs of online-to-offline commerce. Evaluating five state-of-the-art LLM agents against a simple rule-based inventory-control baseline, the authors find that four agents—including Claude Sonnet 5.5, DeepSeek V4.1 Flash, Kimi K3, and GLM-5.3-Flash—underperform the baseline. Only GPT-6.1 Sol consistently exceeds it. The paper matters because it reveals that current LLM agents struggle with the long-horizon coordination, execution planning, and strategy revision required for real-world retail operations, and that gains in individual metrics do not ensure economic growth.
What carries the argument. The central object is the AutoVend simulator, which grounds customer demand in real vending sales records through a multinomial-logit choice model with a no-purchase option, overdispersion, and persistent shocks. The simulator connects a shared warehouse with 1,372 products to 20 machines across five site types (worksite, campus, residential, commercial, office), each with capacity 200 units. Key constraints include a three-day procurement lead time, paid on-site visits (35 CNY per machine per day), inventory expiration at 30 or 90 days, storage costs, and a 1,000-CNY fine for selling expired products. Agents interact via 18 MCP tools for remote operations, physical execution, and persistent memory. The rule-based baseline combines the newsvendor critical ratio with periodic order-up-to replenishment, setting inventory targets from rolling 30-day demand estimates.
Core claim. AutoVend is a benchmark that places an LLM agent in charge of a network of 20 vending machines with heterogeneous demand, a shared warehouse, procurement lead times, paid on-site replenishment, and inventory expiration. Over 180 simulated days, the rule-based baseline achieves mean terminal net assets of 210,970 CNY, while four of the five tested LLM agents fall short: GPT-6.1 Sol reaches 309,350 CNY (the only one to beat the baseline), followed by GLM-5.3-Flash at 148,430 CNY, DeepSeek V4.1 Flash at 133,920 CNY, Claude Sonnet 5.5 at 103,040 CNY, and Kimi K3 at 91,520 CNY. Operational diagnostics show that no single model excels across all five dimensions (profitability, assortment, inventory waste, pricing regret, capital efficiency), and improvements in individual metrics often fail to translate into stronger economic growth due to offsetting inventory and execution costs. Trajectory analysis reveals difficulties in grounding feasible actions, managing inventory commitments, and converting feedback into effective policy revisions.
The premise it rests on. The demand model's price and environmental responses are unvalidated against real data, so the realism of pricing decisions and weather effects on demand is not confirmed, which could bias the evaluation toward or away from certain agent strategies.
If the paper is right (Pith's inferences, not the authors')
If this result holds broadly, deploying LLM agents for real physical retail without significant safeguards or hybrid rule-based components is premature; a simple algorithm already outperforms most agents.
API efficiency (net-asset growth per dollar of inference cost) ranks models differently than raw returns: DeepSeek V4.1 Flash is 28× more cost-efficient than GPT-6.1 Sol, suggesting cost-aware selection matters for deployment.
The failure of agents to convert feedback into sustained policy improvements indicates that continual-learning mechanisms need to be explicitly integrated into LLM agent designs.
Gains in individual operational metrics (e.g., low waste, high assortment quality) do not guarantee aggregate profitability; inventory and execution costs can negate local improvements, so holistic evaluation is essential.
AutoVend could serve as a template for other online-to-offline domains (e.g., food-truck fleets, laundromat networks) where physical execution costs dominate, enabling systematic comparison of agent architectures.
The weak performance of agents on coordination and planning suggests that LLMs lack an internal world model capable of simulating the delayed consequences of inventory decisions; adding a lightweight simulator inside the agent loop might close the gap.
The benchmark's open-source release allows the community to test counterfactuals—for instance, whether fine-tuning on operational feedback traces or augmenting the prompt with explicit inventory-cost calculators improves outcomes.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Large language model (LLM) agents are extending agent-mediated commerce on the Web from shopping for consumers to operating businesses for sellers. In online-to-offline (O2O) commerce, decisions are issued through Web-based platforms but realized through costly physical execution. Evaluating such agents requires going beyond individual transactions to ask whether decisions made online turn into sustained economic returns. We introduce AutoVend, a benchmark in which one LLM agent manages 20 vending machines across five site types through a Web-based operating interface for 180 simulated days. Its data-grounded simulator combines a shared warehouse, procurement lead times, paid replenishment, inventory expiration, and recurring operating costs. We evaluate five LLM agents against a rule-based inventory-control baseline and diagnose their site coordination, execution, and performance over time. Four of five evaluated LLM agents fall short of this baseline, and returns per dollar of inference cost rank models differently from returns alone. Operational analyses show that gains in individual metrics do not consistently yield stronger economic growth, as inventory and execution costs erode these gains. Trajectory analysis further reveals difficulties in grounding feasible actions, managing inventory commitments, and converting feedback into effective policy revisions.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-28
The authors' main claim, as the classifier picked it out 0.56
“Four of five evaluated LLM agents fall short of this baseline, and returns per dollar of inference cost rank models differently from returns alone.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.09 · states limitation 0.67 · human participants 0.04 · controlled comparison 0.98 · randomized assignment 0.02 · states quantitative result 0.93 · compares with prior results 0.96 · mentions code or data availability 0.05
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Large language model (LLM) agents are extending agent-mediated commerce on the Web from shopping for consumers to operating businesses for sellers. | background 1.00 | 0.07 | 0.03 | asserted 1.00 |
| In online-to-offline (O2O) commerce, decisions are issued through Web-based platforms but realized through costly physical execution. | background 0.99 | 0.05 | 0.02 | asserted 1.00 |
| Evaluating such agents requires going beyond individual transactions to ask whether decisions made online turn into sustained economic returns. | gap 0.74 | 0.10 | 0.06 | asserted 0.96 |
| We introduce AutoVend, a benchmark in which one LLM agent manages 20 vending machines across five site types through a Web-based operating interface for 180 simulated days. | contribution 0.88 | 0.95 | 0.04 | asserted 1.00 |
| Its data-grounded simulator combines a shared warehouse, procurement lead times, paid replenishment, inventory expiration, and recurring operating costs. | method 0.98 | 0.78 | 0.03 | asserted 1.00 |
| We evaluate five LLM agents against a rule-based inventory-control baseline and diagnose their site coordination, execution, and performance over time. | method 0.90 | 0.96 | 0.06 | asserted 0.95 |
| Four of five evaluated LLM agents fall short of this baseline, and returns per dollar of inference cost rank models differently from returns alone. | result 1.00 | 0.95 | 0.40 | asserted 1.00 |
| Operational analyses show that gains in individual metrics do not consistently yield stronger economic growth, as inventory and execution costs erode these gains. | interpretation 0.75 | 0.93 | 0.40 | asserted 1.00 |
| Trajectory analysis further reveals difficulties in grounding feasible actions, managing inventory commitments, and converting feedback into effective policy revisions. | result 0.63 | 0.92 | 0.17 | asserted 1.00 |
What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from simulation or computation 0.91
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-28: 1 request, 6,841 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (6)
Figures are the authors', reproduced from the paper.
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment