Pith.

Artificial Intelligence · arXiv:2610.36980 v2 · submitted 27 Oct 2026

AutoVend: Benchmarking LLM Agents on Long-Horizon Autonomous Vending Operations

Qiming Shi, Yikun Li, Beidi Luan, Rui Sun, Sinuo Wang, Delin Wang, Yan Ma, Jing Li

show 2 more authors

Daxin Jiang, Zuo Bai

The paper says

“Large language model (LLM) agents are extending agent-mediated commerce on the Web from shopping for consumers to operating businesses for sellers. In online-to-offline (O2O) commerce, decisions are issued through Web-based platforms but realized through costly physical execution.”

more of the abstract

Evaluating such agents requires going beyond individual transactions to ask whether decisions made online turn into sustained economic returns. We introduce AutoVend, a benchmark in which one LLM agent manages 20 vending machines across five site types through a Web-based operating interface for 180 simulated days. Its data-grounded simulator combines a shared warehouse, procurement lead times, paid replenishment, inventory expiration, and recurring operating costs. We evaluate five LLM agents against a rule-based inventory-control baseline and diagnose their site coordination, execution, and performance over time. Four of five evaluated LLM agents fall short of this baseline, and returns per dollar of inference cost rank models differently from returns alone. Operational analyses show that gains in individual metrics do not consistently yield stronger economic growth, as inventory and execution costs erode these gains. Trajectory analysis further reveals difficulties in grounding feasible actions, managing inventory commitments, and converting feedback into effective policy revisions.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 28 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

The passes disagreed. Reader: holds up. Skeptic: holds up only with conditions. Referee: minor revision.

AutoVend is a well-designed benchmark for seller-side LLM agent evaluation, but its central claim rests on a single random seed and needs broader validation. This line is wrong

2 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at Section 4.1 and Table 2

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.63):

“Future work should test these findings, drawn from three runs per model in one environment, across varied operating conditions and controlled interventions such as explicit state verification and persistent pol- icy updates.”

Pith says: Objection 1 of 2 · major

The central quantitative result that four of five LLM agents underperform the rule-based baseline is based on a single demand seed (all runs use 'one frozen world with the same random seed'). While the baseline is reported to be stable across seeds 1–3 (CV 1.1%), the LLM agents exhibit substantial variance (e.g., Kimi K3 CV 45.6%).

more of objection 1

Without testing over multiple demand realizations, the robustness of the ranking—and of the headline claim itself—is unknown. The paper acknowledges this as future work (sentence b222), but it is a load-bearing limitation for the paper's main conclusion. I recommend either (a) running experiments on at least 2–3 additional seeds with a representative subset of agents (e.g., the best and the worst) to assess rank stability, or (b) prominently qualifying the claim in the abstract and conclusion as preliminary and seed-dependent.

The paper says, at Section 3.3

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.91):

“Price and environmental responses remain unvalidated by design. 5 100 101 102 103 Simulated units 100 101 102 103 Realized units Worksite Campus Residential Commercial Office Unknown TeaWaterEnergySodaSportsJuiceMilkJerkyNoodlePastryCoffeeCandyCookieOther 0% 5% 10% 15% 20% 25% 30%Share of units sold Real Sim (a) Weekly sales per machine (b) Product mix Figure 3: Held-out evaluation of the demand model on 67 machines over an eight-week test window, averaged over 100 seeds.”

Pith says: Objection 2 of 2 · major

The paper states that price and environmental responses 'remain unvalidated by design' (sentence b110). Since agents make pricing decisions and respond to weather/holidays, the simulation's accuracy for these components is unknown. While the demand volume and product mix are validated, the unvalidated price elasticities and weather multipliers could disproportionately affect agents that rely on dynamic pricing or weather-aware strategies.

more of objection 2

I recommend adding a discussion of the potential direction of bias this introduces and, if feasible, a sensitivity analysis over plausible ranges for the price elasticities and environmental response parameters.

6 minor objections

The paper says, at Abstract

Pith says: minor objection 1 of 6

The phrase 'returns per dollar of inference cost rank models differently from returns alone' is interesting but could be misinterpreted. Clarify that the ranking by API efficiency (net-asset growth per USD of API cost) differs from the ranking by terminal net assets.

The paper says, at Table 2

Pith says: minor objection 2 of 6

The table reports 'Shelf Utilization' as a percentage (e.g., 88.91%), but the text later uses a different metric: the 'share of site-days with empty shelves' (e.g., 25.21% for Claude). Consider adding the empty-site-days fraction directly to the table or providing a cross-reference to make the two inventory-availability metrics easier to compare.

show 4 more

The paper says, at Figure 5(b)

Pith says: minor objection 3 of 6

The reassignment gap is defined as a percentage, but the denominator is not explicitly stated in the caption. Clarify how the gap is computed—what is the range and what does 0% correspond to? This would improve interpretability.

The paper says, at Section 4.1 (Baseline description)

Pith says: minor objection 4 of 6

The rule-based baseline is described as using newsvendor critical ratios and periodic order-up-to replenishment, with prices 'adjusted through local gross-profit trials.' Please provide a brief specification of the price-adjustment mechanism (e.g., how often, what step size) to allow reproducibility.

The paper says, at Section 5.2

Pith says: minor objection 5 of 6

The observation that 'all 15 LLM executions finish above the initial capital' is a notable positive result, but it is not mentioned in the conclusion. Consider highlighting it as evidence that even underperforming agents can generate returns, even if they fail to match the baseline.

The paper says, at Throughout

Pith says: minor objection 6 of 6

Minor: ensure consistency in model naming—'GPT-6.1 Sol' vs 'GPT-6.1 Sol' is consistent; double-check capitalization of model names (e.g., 'DeepSeek V4.1 Flash' appears correctly). No typos were found in the reviewed text.

What would settle it

Run every LLM agent (and the rule-based baseline) on each of at least 5 independent world seeds (e.g., seeds 1–5), using 3 LLM runs per seed. Compute the fraction of (seed, LLM-run) pairs where each LLM's terminal net assets exceed the baseline's mean across its own 3 runs per seed. If for any LLM that fraction exceeds 0.5 on average, the claim that it 'underperforms' is not robust across world seeds.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 28 Oct 2026

  1. Reader holds up confidence high
  2. Skeptic disagrees: says it holds up only with conditions 36 s · 18,986 tokens read, 3,583 written Main result rests on a single world seed; generalizability of the 'four of five underperform' claim is untested.
  3. Editor wrote the verdict above 12 s · 19,842 tokens read, 1,190 written
  4. Referee minor revision: 2 major objections, 6 minor 32 s · 19,418 tokens read, 3,933 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

You should know two things about this paper. First, AutoVend is a solidly built benchmark for long-horizon vending operations: 20 heterogeneous machines, shared inventory, procurement lead times, paid site visits, and a demand model grounded in real sales records. The validation against held-out data is credible (WAPE 0.473, TVD 0.153), and the operational diagnostics (assortment, waste, pricing regret, capital efficiency) are thorough. The paper does what a good benchmark paper should: it builds a plausible environment, checks that it behaves like real data, and then runs agents. Second, the headline result—four of five LLM agents underperform a rule-based baseline—is interesting but not yet robust. The experiments use a single world seed, with only three runs per LLM to capture agent stochasticity. The authors acknowledge this in their conclusion, but it remains a load-bearing limitation: we don’t know whether the ranking holds across different demand paths. The paper also states that price and environmental responses are unvalidated by design; given that agents adjust prices and respond to weather, this could affect the realism of the evaluation. These are honest limitations, but they constrain what we can confidently say from the results. The benchmark itself is a real contribution—it fills a gap between single-store simulations and multi-site physical retail. The toolkit and evaluation protocol are reproducible. The paper deserves peer review; the editor should send it. My advice: accept it after the authors add experiments across multiple seeds (or provide a convincing argument that seed variation doesn’t change the qualitative result) and acknowledge the unvalidated components more prominently. This is a useful resource for anyone working on agent-mediated commerce or long-horizon planning.

The strongest objection

The central claim that four of five LLM agents underperform the rule-based baseline (sentence b219) is based on evaluations performed on a single frozen world seed (sentence "all in one frozen world with the same random seed"). Variation across LLM runs captures only agent stochasticity, not variation in the demand path. The baseline's stability across seeds 1–3 (CV 1.1%) does not guarantee that LLM performance relative to the baseline is similarly stable across different demand realizations. Given the high variance of Kimi K3 (CV 45.6%) even on one seed, and the moderate variance of others (~11%), different seeds could shift rankings. The paper acknowledges this only as future work (sentence b222: "drawn from three runs per model in one environment"), but it is a load-bearing limitation because the headline quantitative result is not demonstrated to be robust across the stochasticity that the paper intentionally built into the simulator (overdispersed shocks, auto-correlation, weather, holidays). A claim that a majority of agents underperform a baseline should be tested over multiple demand paths to establish its reliability.

The reader identified the unvalidated price/environmental responses as the weakest assumption, but that concern targets realism rather than the reliability of the central comparative result within the simulation framework. The more immediately load-bearing concern for the paper's main claim is that the comparison rests on a single random seed. The authors acknowledge the limitation but do not mitigate it in the present work. While the benchmark and diagnostics are still valuable, the headline result that 'four of five underperform' is not yet established as a robust finding. A conditional verdict—acceptance contingent on multi-seed validation—is appropriate.

The referee's report

AutoVend fills an important gap in seller-side agent evaluation by combining multi-site coordination, physical execution costs, and long-horizon economic returns. The simulator is grounded in real vending sales records and validated on demand volume and product mix. The finding that most evaluated LLM agents fail to beat a straightforward algorithmic baseline is striking and provides a clear, reproducible challenge for future work on long-horizon autonomous commerce agents. The benchmark is well-designed, the evaluation protocol is thorough, and the diagnostics offer insight beyond a single aggregate metric. The paper is likely to be influential in the agent-mediated commerce community.

Pith's reading of the paper (a machine's summary, not the authors')

The paper introduces AutoVend, a benchmark where a single LLM agent manages 20 vending machines across five site types for 180 simulated days. The environment includes shared inventory, procurement lead times, paid replenishment, and product expiration, reflecting the physical costs of online-to-offline commerce. Evaluating five state-of-the-art LLM agents against a simple rule-based inventory-control baseline, the authors find that four agents—including Claude Sonnet 5.5, DeepSeek V4.1 Flash, Kimi K3, and GLM-5.3-Flash—underperform the baseline. Only GPT-6.1 Sol consistently exceeds it. The paper matters because it reveals that current LLM agents struggle with the long-horizon coordination, execution planning, and strategy revision required for real-world retail operations, and that gains in individual metrics do not ensure economic growth.

What carries the argument. The central object is the AutoVend simulator, which grounds customer demand in real vending sales records through a multinomial-logit choice model with a no-purchase option, overdispersion, and persistent shocks. The simulator connects a shared warehouse with 1,372 products to 20 machines across five site types (worksite, campus, residential, commercial, office), each with capacity 200 units. Key constraints include a three-day procurement lead time, paid on-site visits (35 CNY per machine per day), inventory expiration at 30 or 90 days, storage costs, and a 1,000-CNY fine for selling expired products. Agents interact via 18 MCP tools for remote operations, physical execution, and persistent memory. The rule-based baseline combines the newsvendor critical ratio with periodic order-up-to replenishment, setting inventory targets from rolling 30-day demand estimates.

Core claim. AutoVend is a benchmark that places an LLM agent in charge of a network of 20 vending machines with heterogeneous demand, a shared warehouse, procurement lead times, paid on-site replenishment, and inventory expiration. Over 180 simulated days, the rule-based baseline achieves mean terminal net assets of 210,970 CNY, while four of the five tested LLM agents fall short: GPT-6.1 Sol reaches 309,350 CNY (the only one to beat the baseline), followed by GLM-5.3-Flash at 148,430 CNY, DeepSeek V4.1 Flash at 133,920 CNY, Claude Sonnet 5.5 at 103,040 CNY, and Kimi K3 at 91,520 CNY. Operational diagnostics show that no single model excels across all five dimensions (profitability, assortment, inventory waste, pricing regret, capital efficiency), and improvements in individual metrics often fail to translate into stronger economic growth due to offsetting inventory and execution costs. Trajectory analysis reveals difficulties in grounding feasible actions, managing inventory commitments, and converting feedback into effective policy revisions.

The premise it rests on. The demand model's price and environmental responses are unvalidated against real data, so the realism of pricing decisions and weather effects on demand is not confirmed, which could bias the evaluation toward or away from certain agent strategies.

If the paper is right (Pith's inferences, not the authors')

If this result holds broadly, deploying LLM agents for real physical retail without significant safeguards or hybrid rule-based components is premature; a simple algorithm already outperforms most agents.

API efficiency (net-asset growth per dollar of inference cost) ranks models differently than raw returns: DeepSeek V4.1 Flash is 28× more cost-efficient than GPT-6.1 Sol, suggesting cost-aware selection matters for deployment.

The failure of agents to convert feedback into sustained policy improvements indicates that continual-learning mechanisms need to be explicitly integrated into LLM agent designs.

Gains in individual operational metrics (e.g., low waste, high assortment quality) do not guarantee aggregate profitability; inventory and execution costs can negate local improvements, so holistic evaluation is essential.

AutoVend could serve as a template for other online-to-offline domains (e.g., food-truck fleets, laundromat networks) where physical execution costs dominate, enabling systematic comparison of agent architectures.

The weak performance of agents on coordination and planning suggests that LLMs lack an internal world model capable of simulating the delayed consequences of inventory decisions; adding a lightweight simulator inside the agent loop might close the gap.

The benchmark's open-source release allows the community to test counterfactuals—for instance, whether fine-tuning on operational feedback traces or augmenting the prompt with explicit inventory-cost calculators improves outcomes.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Large language model (LLM) agents are extending agent-mediated commerce on the Web from shopping for consumers to operating businesses for sellers. In online-to-offline (O2O) commerce, decisions are issued through Web-based platforms but realized through costly physical execution. Evaluating such agents requires going beyond individual transactions to ask whether decisions made online turn into sustained economic returns. We introduce AutoVend, a benchmark in which one LLM agent manages 20 vending machines across five site types through a Web-based operating interface for 180 simulated days. Its data-grounded simulator combines a shared warehouse, procurement lead times, paid replenishment, inventory expiration, and recurring operating costs. We evaluate five LLM agents against a rule-based inventory-control baseline and diagnose their site coordination, execution, and performance over time. Four of five evaluated LLM agents fall short of this baseline, and returns per dollar of inference cost rank models differently from returns alone. Operational analyses show that gains in individual metrics do not consistently yield stronger economic growth, as inventory and execution costs erode these gains. Trajectory analysis further reveals difficulties in grounding feasible actions, managing inventory commitments, and converting feedback into effective policy revisions.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-28

The authors' main claim, as the classifier picked it out 0.56

“Four of five evaluated LLM agents fall short of this baseline, and returns per dollar of inference cost rank models differently from returns alone.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.09 · states limitation 0.67 · human participants 0.04 · controlled comparison 0.98 · randomized assignment 0.02 · states quantitative result 0.93 · compares with prior results 0.96 · mentions code or data availability 0.05

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Large language model (LLM) agents are extending agent-mediated commerce on the Web from shopping for consumers to operating businesses for sellers. background 1.00 0.07 0.03 asserted 1.00
In online-to-offline (O2O) commerce, decisions are issued through Web-based platforms but realized through costly physical execution. background 0.99 0.05 0.02 asserted 1.00
Evaluating such agents requires going beyond individual transactions to ask whether decisions made online turn into sustained economic returns. gap 0.74 0.10 0.06 asserted 0.96
We introduce AutoVend, a benchmark in which one LLM agent manages 20 vending machines across five site types through a Web-based operating interface for 180 simulated days. contribution 0.88 0.95 0.04 asserted 1.00
Its data-grounded simulator combines a shared warehouse, procurement lead times, paid replenishment, inventory expiration, and recurring operating costs. method 0.98 0.78 0.03 asserted 1.00
We evaluate five LLM agents against a rule-based inventory-control baseline and diagnose their site coordination, execution, and performance over time. method 0.90 0.96 0.06 asserted 0.95
Four of five evaluated LLM agents fall short of this baseline, and returns per dollar of inference cost rank models differently from returns alone. result 1.00 0.95 0.40 asserted 1.00
Operational analyses show that gains in individual metrics do not consistently yield stronger economic growth, as inventory and execution costs erode these gains. interpretation 0.75 0.93 0.40 asserted 1.00
Trajectory analysis further reveals difficulties in grounding feasible actions, managing inventory commitments, and converting feedback into effective policy revisions. result 0.63 0.92 0.17 asserted 1.00

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from simulation or computation 0.91

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-28: 1 request, 6,841 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (6)
Figure 1 of the paper
Figure 1. AutoVend connects Web-based agent deci￾sions to distributed physical retail through shared inven￾tory, procurement lead times, and paid on-site replenish￾ment. depend on the costs and constraints of physical execution. For agents managing distributed retail through such platforms, this requires coordinat￾ing digital decisions with physical execution over time. Smart vending is a compact instance, as connected machines are increasingly run through cloud management platforms (Fortune Business In￾sights, 2026) while every restock requires a paid site visit. It also retains the…
Figure 2 of the paper
Figure 2. Overview of AutoVend. (a) Agent and interface, with MCP tools for remote operations, physical execution,
Figure 3 of the paper
Figure 3. Held-out evaluation of the demand model on 67 machines over an eight-week test window, averaged
Figure 4 of the paper
Figure 4. Whole-episode operational profiles over 180 days. Each panel shows the mean operational profile of an
Figure 5 of the paper
Figure 5. Long-horizon operations, site coordination, and physical execution. (a) Mean net-asset trajectories across
Figure 6 of the paper
Figure 6. Early- and late-period operating outcomes.

Figures are the authors', reproduced from the paper.

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment