Pith.

Artificial Intelligence · arXiv:2607.28956 v2 · submitted 31 Jul 2026 · 34 references

MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations

Qiming Shi, Yulong Tao, Linbo Jin, Zhaolu Kang, Yibo Dou, Jiawen Zhu, Tianjun Pan, Shaokang Fu

show 5 more authors

Chengyu Wang, Siyue Li, Yaping Cheng, Di Weng, Chengfu Huo

The paper says

“Large language model agents are increasingly evaluated as autonomous tool users, yet most benchmarks focus on bounded tasks with immediate success criteria.”

more of the abstract

Real-world deployments often require Long-Term Coherence, the capacity to preserve purposeful behavior across extended horizons while adapting decisions to accumulated evidence. Evaluating this capacity requires a persistent environment in which actions constrain future choices, feedback arrives at heterogeneous delays, and incoherent behavior produces measurable cumulative effects. Seller-side e-commerce provides a suitable setting for this evaluation through recurrent and interdependent decisions over Product Sourcing, Listing and Pricing Control, Cash-Flow Management, and Mixed-Latency Feedback Adaptation. We introduce MerchantBench, a 365-day order-level simulation grounded in 98,843 real e-commerce product records and equipped with 26 tools for agent interaction. MerchantBench couples promptly observable Upstream Supplier Events with delayed Downstream Order Outcomes, requiring agents to follow individual order lifecycles and revisit earlier decisions. We evaluate eight LLMs under two agent frameworks in 48 runs, each spanning 365 simulated days. Our results reveal a substantial gap between even the latest LLMs and human participants, with the best LLM configuration attaining only 27.3% of the mean final net assets achieved by human participants.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 5 Aug 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Genuinely useful benchmark with unusually full disclosure, but the headline 27.3% human-gap number rests on a 3-person baseline and a context-compression confound the paper never isolates. This line is wrong

4 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 34 references say.

The paper says, at §Detailed Experimental Configuration (Context Management); §Long-Term Coherence Analysis

The review did not quote this passage.

Pith says: Objection 1 of 4 · major

Both agent frameworks force lossy self-summarization: ReAct summarizes at 160k tokens and truncates to the most recent 30k, and Hermes uses its default summarizer with the evaluated model as its own summarizer. The paper reports no run without this compression and no analysis of summary fidelity.

more of objection 1

The taxonomy's flagship examples — the Hermes Kimi K2.6 run that judged the store unrecoverable on Day 104 and stopped acting in 355 of 523 windows, and the Hermes Qwen3.7-Max run that misremembered Day 285 as the endpoint on Day 282 — are exactly the kind of outcomes that lossy summary/truncation would produce. The fact that GPT-5.6 Sol sustains SWR 99.4% under the same ReAct protocol, and that windows display calendar time, shows the protocol alone does not force collapse, but it does not isolate the variable. Without a no-compression or ground-truth-summary condition, the headline gap and tax

The paper says, at §Experimental Setup (Evaluation Protocol); Table 1

The review did not quote this passage.

Pith says: Objection 2 of 4 · major

The headline ratio (best LLM 27.3% of human mean final net assets, abstract; Table 1) rests on a human baseline of three participants, each completing one run, with no prior e-commerce experience and no reported per-participant results, variance, or significance test.

more of objection 2

The human interface is described only as a "human operations dashboard" (Detailed Experimental Configuration, Evaluation Protocol), so interface comparability with the 26-tool LLM action space is unestablished. LLM runs are themselves highly variable (Qwen3.7-Max/Hermes CV = 55.1%, Fig. 4). The paper should report individual human trajectories, variability, and an appropriate test or interval, and should specify the dashboard's information and action affordances. As presented, the central gap cannot be distinguished from sampling variation.

The paper says, at §Real-World Data Grounding; Eqs. (3)-(5); Detailed Experimental Configuration

The review did not quote this passage.

Pith says: Objection 3 of 4 · major

Several environment parameters that drive the measured outcomes are set by fiat rather than estimated from the grounding data: price elasticities ε_i in Eq. (3), exposure parameters ℓ0, Tr, κ, ℓmin in Eq. (4), rating evidence scores u_j/weights v_j, thresholds, and demand multipliers in Eq.

more of objection 3

(5) and the appendix. The qualitative conclusions — e.g., the positive associations in Figs. 8 and 10 — may be sensitive to these choices. A sensitivity analysis over the most influential parameters (or a clear calibration source for each) is needed to establish that the coherence-failure pattern is a property of the setting, not of one parameter vector. Currently the benchmark instantiates one point in a large parameter family.

The paper says, at §Long-Term Coherence Analysis

The review did not quote this passage.

Pith says: Objection 4 of 4 · major

The Long-Term Coherence taxonomy is introduced in the paper and then used to interpret traces, but no coding protocol, inter-annotator reliability, or quantitative prevalence counts are given. Claims such as "ReAct Qwen3.7-Max ... supply chain checks rising from 14% to 34% of remaining tool calls" and the product-level risk-response groupings are based on representative runs.

more of objection 4

Since the taxonomy is a core contribution, the authors should provide a trace-coding scheme, report how many of the 48 runs exhibit each failure mode, and quote or tabulate representative evidence; otherwise the taxonomy is unfalsifiable from the reported results.

5 minor objections

The paper says, at Table 1

Pith says: minor objection 1 of 5

Values run together in several cells (e.g., "7964.0512.1", "2,2301,474"); reformat the table. Also state in the caption which numbers are means across three runs vs. single human runs.

The paper says, at Figures

Pith says: minor objection 2 of 5

Figures 9 and 10 are referenced in the main text ahead of the appendix figures 11-18; renumber or reorder so the first-mention order is consistent.

show 3 more

The paper says, at §Full Monthly Product Sourcing Analysis, Eqs. (6)-(8)

Pith says: minor objection 3 of 5

The "no-reallocation counterfactual" in Eq. (7) uses the run's own annual listing hours W_{r,p}, so it is a descriptive decomposition rather than a causal counterfactual; qualify the terminology.

The paper says, at Abstract / Experimental Setup

Pith says: minor objection 4 of 5

The phrase "best LLM configuration" is a selection over 16 configurations; state the selection criterion (mean over three runs) explicitly.

The paper says, at Detailed Experimental Configuration, Evaluation Protocol

Pith says: minor objection 5 of 5

State whether the human dashboard showed the same observation cadence and summary format as the agents; this is needed to interpret the human/LLM comparison.

What would settle it

Run an ablation on the highest-decay configurations (Hermes Qwen3.7-Max, Hermes Kimi K2.6, Hermes Claude Opus 4.8, ReAct Qwen3.7-Max), three runs each, holding environment, prompts, tools, and compute budgets fixed, but at each compression event replacing the model-authored memory with an environment-written lossless structured state record (remaining horizon, cash, deposit, active listings, open orders, recent anomalies, pending strategies). If Day-104-style abandonment, the Day-285 horizon error, and SWR collapse vanish or shrink, the coherence taxonomy measures harness artifacts; if decay persists, the taxonomy stands. A cheaper check on released logs: for each flagged failure, count prior compression events and inspect stored memory at the failure window for horizon and recovery evidence.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 5 Aug 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 3 min 1 s · 1,307 tokens read, 2,559 written
  3. Editor wrote the verdict above 36 s · 26,797 tokens read, 3,249 written
  4. Referee major revision: 4 major objections, 5 minor 1 min 28 s · 26,154 tokens read, 8,241 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Short version: this is a real benchmark contribution, carefully specified and honestly reported in most places, but the abstract's headline number overstates what the evidence supports. The environment deserves serious referee attention; the empirical gap claim needs another round of work.

What's actually new: the combination of order-level lifecycle simulation with presampled hidden outcomes, asymmetric feedback latency (prompt supplier events vs. delayed order outcomes), a 98,843-product catalog with real 365-day demand traces, and a 365-day horizon under two frameworks. I don't know of another environment that puts these together. The disclosure level is unusually high: all 26 tools, both system prompts, full parameter values, 18 figures, released code. That is the kind of transparency that makes a benchmark useful even when its conclusions are contested. The failure taxonomy (activity decay, premature abandonment, goal drift, miscalibrated adaptation) is a reasonable way to organize the traces.

Soft spots, in order of importance. First, the human baseline: three participants, one run each, no prior e-commerce experience, no variance reported. The 27.3% figure is the mean of three people. And the "best LLM" is best-of-16 configurations, the winning one with a 55.1% coefficient of variation, so the headline gap has a wide error bar on both sides. Second, and more interesting: the paper attributes coherence failures to the LLMs, but both frameworks force lossy self-summarization—ReAct summarizes at 160k tokens and truncates to 30k, Hermes uses its default summarizer with the evaluated model as its own summarizer. The paper's own exhibits, the Kimi run that gave up on Day 104 and the Qwen run that misremembered Day 285 as the endpoint on Day 282, sit exactly where compression artifacts should bite. The construct-validity threat is real: the taxonomy may describe model-plus-truncation rather than model capacity. To be fair, the appendix does disclose the compression settings, and GPT-5.6 Sol sustains a 99.4% SWR under the same ReAct protocol, so compression alone does not force decay. But the paper never flags the variable as a potential confound and runs no ablation on it. Third, smaller: the "calibrated" risk rates are asserted without a disclosed calibration procedure, and proprietary data plus closed APIs limit independent replication. Also, the activity-causes-assets association is partly built into the environment, since an idle store stops adapting to changing demand.

None of this breaks the environment. The core value—a persistent, cash-constrained, order-level testbed with mixed-latency feedback and real demand data—holds up. The central claim is conditional, not wrong: if the compression confound is resolved, the gap and taxonomy stand.

This paper is for people building long-horizon agent benchmarks and for anyone evaluating LLM agents who wants a model of how to specify and disclose an environment. Send it to peer review. Ask for either an ablation that isolates the compression schedule or an explicit reframing of compression as part of the measured system, plus variance information or additional human runs. As it stands, it's a good benchmark attached to a headline that outruns its evidence.

The strongest objection

The central claim - that LLM agents lose Long-Term Coherence over 365 days, with the paper's taxonomy (Operational Coherence: activity decay, premature abandonment; Strategic Coherence: goal drift, miscalibrated adaptation) characterizing the evaluated models - assumes the context-management protocol is not the cause. The appendix (Detailed Experimental Configuration, Context Management) discloses that both frameworks compress: ReAct triggers summarization at 160k tokens and truncates to the most recent 30k; Hermes uses its default summarizer, with each evaluated model summarizing its own history. No run without this compression and no analysis of summary fidelity is reported. The paper's own exhibits sit exactly on this fault line: the Hermes Kimi K2.6 run that judged the store unrecoverable on Day 104 and then took no action in 355 of 523 remaining windows, and the Hermes Qwen3.7-Max run that misremembered Day 285 as the endpoint on Day 282 and stopped filling vacant slots with 83 days left, are signature outcomes of self-summarization. The Hermes Claude Opus 4.8 false inference about traffic concentration likewise rests on unexamined memory traces. This is a construct-validity threat: if compression drives the decay, the headline gap and taxonomy describe the model-plus-truncation system, not LLM capacity. Countervailing evidence: GPT-5.6 Sol sustains SWR 99.4% under the same ReAct protocol, and windows show the calendar day, so the endpoint error is partly model-side. Still, the paper never flags compression as a potential confound, so the benchmark cannot separate 'model loses coherence' from 'fixed compression schedule corrupts state representation.'

The benchmark's contribution is a measurement instrument plus a diagnosis: LLM agents fall far short of humans (best configuration 27.3% of mean human final net assets, Table 1) because of measurable Long-Term Coherence failures. For that diagnosis to hold, the failures must be properties of the evaluated agents, not of the harness that lets them operate across 730 decision windows. The disclosed context-management protocol is the largest unablated component of that harness: every agent is forced, at framework-set thresholds, to rebuild its own state representation through lossy self-summarization. The paper's marquee anecdotes - Day-104 abandonment, Day-285 misremembered endpoint, locked-in false beliefs about traffic concentration - are precisely the behaviors lossy self-summarization would produce, and the paper presents them as evidence of model-level incoherence without testing the alternative. I judge this more load-bearing than other weaknesses: the human baseline (n=3, unreported variance) threatens precision of the 27.3% figure but not its qualitative existence given the 3.7x margin; undisclosed risk calibration threatens external validity only. The compression confound threatens construct validity - the very construct the benchmark claims to measure. The concern is not airtight, and the paper deserves credit for disclosing the protocol and releasing code: GPT-5.6 Sol maintains SWR 99.4% under the same ReAct truncation, and the observation includes the current calendar day, so at least the endpoint error is partly model-side. This is why the right output is a demanding condition (isolate compression) rather than rejection. The reader's CONDITIONAL already encodes this; my pass confirms the compression variable is the single most load-bearing of the flagged weaknesses, identifies exactly which reported behaviors it would explain, and supplies a settling test. Verdict: UNCHANGED.

The referee's report

If the central result holds, MerchantBench addresses a genuinely underexplored evaluation dimension — sustained coherent behavior over long horizons with mixed-latency feedback — and its failure taxonomy would be a useful diagnostic. The manuscript's strengths are concrete: it ships code, provides the complete 26-tool interface, both system prompts, all parameter values, and 18 figures, and its metrics (SWR, monthly demand alignment, time-aware sourcing gain) are well-defined and falsifiable. The principal risk is construct validity: the measured decay may be an artifact of the forced lossy context compression rather than of the models themselves, and the human baseline is too thinly reported to anchor the headline gap. If the authors address these points, the benchmark could be a valuable community resource.

Pith's reading of the paper (a machine's summary, not the authors')

This paper tries to establish that long-term coherence — the ability to keep pursuing a goal purposefully across an extended horizon while adapting to evidence that arrives with delays — is a real, measurable capacity that separates current LLM agents from humans. To show this, it builds MerchantBench, a 365-day order-level e-commerce simulation grounded in 98,843 real product records, and runs eight LLMs under two agent frameworks alongside human operators and a rule-based baseline. The central result is a large gap: the best LLM configuration attains only 27.3% of the mean final net assets achieved by humans (RMB 59.46k versus RMB 217.61k). Trace analysis attributes the gap to two failure families: operational incoherence (activity decay, premature abandonment) and strategic incoherence (goal drift, evidence-miscalibrated adaptation). If true, MerchantBench gives the field a way to measure a capability that bounded-task benchmarks cannot see, and names concrete failure modes to target.

What carries the argument. Two coupled mechanisms carry the argument. The first is the mixed-latency feedback structure: Upstream Supplier Events (price change, product delisting, shipment delay) are promptly observable, while Downstream Order Outcomes (cancellation, stockout, late shipment, refunds, bad review) surface only through each order's lifecycle — placed, procured, shipped, delivered, settled — forcing the agent to revisit earlier listing and pricing decisions. The second is the measurement scaffold: business performance (final net assets = cash + deposit + funds in transit + receivables), store reliability (fines, rating, anomaly rate), and long-horizon activity anchored by Sustained Window Rate (SWR), the

Core claim. MerchantBench is a 365-day, order-level simulation of seller-side e-commerce, cast as a finite-horizon POMDP with 8,760 hourly steps, a decision window every 12 hours, 26 tools, and a catalog of 98,843 real products. The central claim: LLMs evaluated here systematically lose coherence over the horizon. In 48 runs across eight LLMs and two frameworks, the best configuration (Qwen3.7-Max under Hermes) reached mean final net assets of RMB 59.46k, 27.3% of the human mean (RMB 217.61k). Traces attribute the gap to Operational Coherence failures — activity decay and premature abandonment (one Kimi K2.6 run gave up on Day 104, acting in none of 355 remaining windows) — and Strategic Coherence failu

The premise it rests on. The paper attributes the coherence failures to the LLMs themselves, assuming the context-management protocol does not cause them: both frameworks compress long histories with each model as its own summarizer (ReAct drops everything but the most recent 30,000 tokens once a run reaches 160,000), so activity decay, premature abandonment, and the misremembered horizon could be artifacts of lossy memory rather than genuine long-horizon incoherence.

If the paper is right (Pith's inferences, not the authors')

MerchantBench becomes a reusable instrument: any agent design can be scored on a 365-day persistent horizon through a 26-tool interface, with a human reference line (100% SWR, mean net assets RMB 217.61k) to measure against.

The failure taxonomy gives concrete engineering targets: activity decay, premature abandonment, control-loop narrowing, goal drift, and evidence-miscalibrated adaptation are now named, traceable failure modes for future agent architectures.

Framework choice helps but does not close the gap: Hermes raised mean net assets 53.3% and orders 71.2% over ReAct averaged across models, yet benefits are model-dependent (Kimi K2.6 did worse under Hermes), and the best Hermes run still reached only 27.3% of the human mean.

Sustained activity alone is insufficient: more tool calls and broader product exploration correlate with higher final net assets, but the dispersion shows activity must be converted into profitable decisions — the coherence problem is about what agents do with each window, not just whether they act.

Humans earn their advantage partly by reallocating the portfolio with the season — their monthly demand-alignment percentile rises from 56.1 to above 80 — and the paper reports that stronger seasonal reallocation (Time-aware Sourcing Gain) accompanies higher final net assets.

My inference: the same decay likely generalizes beyond retail to any long-horizon deployment with delayed consequences — finance, logistics, operations — because activity decay, abandonment, and goal drift are failure modes of lossy-memory agents in persistent environments, not of e-commerce specifically.

My inference: the paper never isolates the context-management protocol from the models — ReAct summarizes at 160k tokens and truncates to the most recent 30k, with each model acting as its own summarizer — so the measured gap may describe the agent-plus-compression system rather than the LLM alone; changing the memory architecture could shrink the gap without any model improvement.

My testable extension: correlate abandonment and decay onset with history-truncation events in the run logs; if decay clusters immediately after summarization, the cause is memory loss, and if it appears with ample context, it is a genuine long-horizon property.

My inference: the positive association between Time-aware Sourcing Gain and final net assets suggests seasonal portfolio reallocation is a learnable, metric-driven skill, and a natural next step is using it as a reward signal for training or fine-tuning agents.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper · Review history (2 versions)

The paper

Abstract

Large language model agents are increasingly evaluated as autonomous tool users, yet most benchmarks focus on bounded tasks with immediate success criteria. Real-world deployments often require Long-Term Coherence, the capacity to preserve purposeful behavior across extended horizons while adapting decisions to accumulated evidence. Evaluating this capacity requires a persistent environment in which actions constrain future choices, feedback arrives at heterogeneous delays, and incoherent behavior produces measurable cumulative effects. Seller-side e-commerce provides a suitable setting for this evaluation through recurrent and interdependent decisions over Product Sourcing, Listing and Pricing Control, Cash-Flow Management, and Mixed-Latency Feedback Adaptation. We introduce MerchantBench, a 365-day order-level simulation grounded in 98,843 real e-commerce product records and equipped with 26 tools for agent interaction. MerchantBench couples promptly observable Upstream Supplier Events with delayed Downstream Order Outcomes, requiring agents to follow individual order lifecycles and revisit earlier decisions. We evaluate eight LLMs under two agent frameworks in 48 runs, each spanning 365 simulated days. Our results reveal a substantial gap between even the latest LLMs and human participants, with the best LLM configuration attaining only 27.3% of the mean final net assets achieved by human participants.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 1.00

“Our results reveal a substantial gap between even the latest LLMs and human participants, with the best LLM configuration attaining only 27.3\% of the mean final net assets achieved by human participants.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.09 · states limitation 0.34 · human participants 0.82 · controlled comparison 0.66 · randomized assignment 0.04 · states quantitative result 0.99 · compares with prior results 0.85 · mentions code or data availability 0.06

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v2). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Large language model agents are increasingly evaluated as autonomous tool users, yet most benchmarks focus on bounded tasks with immediate success criteria. background 0.69 0.05 0.02 asserted 0.99
Real-world deployments often require Long-Term Coherence, the capacity to preserve purposeful behavior across extended horizons while adapting decisions to accumulated evidence. background 0.81 0.06 0.03 asserted 0.98
Evaluating this capacity requires a persistent environment in which actions constrain future choices, feedback arrives at heterogeneous delays, and incoherent behavior produces measurable cumulative effects. background 0.59 0.09 0.02 asserted 1.00
Seller-side e-commerce provides a suitable setting for this evaluation through recurrent and interdependent decisions over Product Sourcing, Listing and Pricing Control, Cash-Flow Management, and Mixed-Latency Feedback Adaptation. method 0.42 0.21 0.02 asserted 0.95
We introduce MerchantBench, a 365-day order-level simulation grounded in 98,843 real e-commerce product records and equipped with 26 tools for agent interaction. contribution 0.86 0.94 0.03 asserted 1.00
MerchantBench couples promptly observable Upstream Supplier Events with delayed Downstream Order Outcomes, requiring agents to follow individual order lifecycles and revisit earlier decisions. method 0.89 0.69 0.03 asserted 1.00
We evaluate eight LLMs under two agent frameworks in 48 runs, each spanning 365 simulated days. method 0.98 0.96 0.02 asserted 0.97
Our results reveal a substantial gap between even the latest LLMs and human participants, with the best LLM configuration attaining only 27.3\% of the mean final net assets achieved by human participants. result 1.00 0.97 0.87 asserted 1.00

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from simulation or computation 0.74

What was examined, and what was not

Sources read

  • arXiv listing abstract v2

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 6,369 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (18)
Figure 1 of the paper
Figure 1. Order-level dynamics couple immediate liquidity
Figure 2 of the paper
Figure 2. Overview of MerchantBench. The merchant agent coordinates four decision components through shorter supplier
Figure 3 of the paper
Figure 3. Real-world demand patterns and calibrated risk
Figure 4 of the paper
Figure 4. Final net asset distributions across three repeated runs for each configuration.
Figure 5 of the paper
Figure 5. Monthly Operational Coherence profiles for the Human baseline and all eight Hermes models. Panels report effective
Figure 6 of the paper
Figure 6. Monthly Product Sourcing across four represen
Figure 7 of the paper
Figure 7. Order scale and unit profitability across 48 LLM
Figure 8 of the paper
Figure 8. Final net assets against environment tool calls and
Figure 9 of the paper
Figure 9. Monthly Product Sourcing across all eight models. Monthly Demand Alignment Percentile is the listing-hour-weighted mean percentile of active products after ranking the complete catalog by that month’s real demand. Lines and bands show repeat means and standard deviations, with Human and Rule-based as shared references
Figure 10 of the paper
Figure 10. Time-aware Sourcing Gain and final net assets.
Figure 11 of the paper
Figure 11. Composition and calibrated distributions of the filtered data. Panel (a) reports the numbers of Product Records and
Figure 12 of the paper
Figure 12. Temporal demand patterns in the filtered data. The top panel presents aggregate daily demand and its seven day mean.
Figure 13 of the paper
Figure 13. Monthly Operational Coherence profiles for the Human baseline and each of the eight Hermes models. Panels (a),
Figure 14 of the paper
Figure 14. Monthly Operational Coherence profiles for the Human baseline and each of the eight ReAct models. Panels (a), (b),
Figure 15 of the paper
Figure 15. Daily net asset curves for all eight models under ReAct over 365 simulated days.
Figure 16 of the paper
Figure 16. Daily net asset curves for all eight models under Hermes over 365 simulated days.
Figure 17 of the paper
Figure 17. Daily net asset curves for the Human and Rule-based baselines and all eight models over 365 simulated days.
Figure 18 of the paper
Figure 18. Daily net asset curves for all individual runs. The first row shows the Human and Rule-based baselines, and the

Figures are the authors', reproduced from the paper.

References (34)
  1. [1] Yao, Shunyu and Shinn, Noah and Razavi, Pedram and Narasimhan, Karthik , booktitle =. 2025 , url = on Pith
  2. [2] AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents , author =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =. 2024 , doi = on Pith
  3. [3] WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents , author =. Advances in Neural Information Processing Systems , volume =. 2022 , doi = on Pith
  4. [4] AgentBench: Evaluating LLMs as Agents , author =. Proceedings of the International Conference on Learning Representations , year = on Pith
  5. [5] WebArena: A Realistic Web Environment for Building Autonomous Agents , author =. Proceedings of the International Conference on Learning Representations , year = on Pith
  6. [6] ECom-Bench: Can LLM Agent Resolve Real-World E-commerce Customer Support Issues? , author =. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track , pages =. 2025 , doi = on Pith
  7. [7] ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents , author =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =. 2026 , doi = on Pith
  8. [8] A Functionality-Grounded Benchmark for Evaluating Web Agents in E-commerce Domains , author =. 2025 , eprint = on Pith
  9. [9] ShopSimulator: Evaluating and Exploring RL-Driven LLM Agent for Shopping Assistants , author =. 2026 , eprint = on Pith
  10. [10] ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents , author =. 2026 , eprint = on Pith
  11. [11] EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent , author =. 2026 , eprint = on Pith
  12. [12] EComStage: Stage-wise and Orientation-specific Benchmarking for Large Language Models in E-commerce , author =. 2026 , eprint = on Pith
  13. [13] Evaluating Multi-Turn Bargain Skills in LLM-Based Seller Agent , author =. 2025 , eprint = on Pith
  14. [14] Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents , author =. 2025 , eprint = on Pith
  15. [15] 2025 , howpublished = on Pith
  16. [16] RetailBench: Evaluating Long-Horizon Autonomous Decision-Making and Strategy Stability of LLM Agents in Realistic Retail Environments , author =. 2026 , eprint = on Pith
  17. [17] CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies , author =. 2026 , eprint = on Pith
  18. [18] EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economies , author =. 2026 , eprint = on Pith
  19. [19] AI Agents for Inventory Control: Human-LLM-OR Complementarity , author =. 2026 , eprint = on Pith
  20. [20] OFCOURSE: A Multi-Agent Reinforcement Learning Environment for Order Fulfillment , author =. Advances in Neural Information Processing Systems , volume =. 2023 , doi = on Pith
show 14 more
  1. [21] Market-Bench: Benchmarking Large Language Models on Economic and Trade Competition , author =. 2026 , eprint = on Pith
  2. [22] Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets , author =. 2025 , eprint = on Pith
  3. [23] 1688: China's Leading Domestic Wholesale Marketplace , year = on Pith
  4. [24] Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle =. 2023 , url = on Pith
  5. [25] Hermes Agent , author =. 2026 , howpublished = on Pith
  6. [26] Planning and Acting in Partially Observable Stochastic Domains , author =. Artificial Intelligence , volume =. 1998 , doi = on Pith
  7. [27] 2026 , howpublished = on Pith
  8. [28] Introducing. 2026 , howpublished = on Pith
  9. [29] UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarios , author =. 2025 , eprint = on Pith
  10. [30] OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows , author =. 2025 , eprint = on Pith
  11. [31] HeroBench: A Benchmark for Long-Horizon Planning and Structured Reasoning in Virtual Worlds , author =. 2025 , eprint = on Pith
  12. [32] Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks , author =. 2026 , eprint = on Pith
  13. [33] WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation , author =. 2026 , eprint = on Pith
  14. [34] Yuan, Mengqi and Zhou, Zilong and Xiong, Xinzhuang and Wu, Weiming and Sun, Jiayang and Song, Jiamin and Cui, Kaiqian and Wang, Bowen and Wu, Haoyuan and Li, Yitong and Lu, Dunjie and Lu, Haikong and Zhen, Qi and Wang, Xinyuan and Deng, Jiaqi and Yang, Yuhao and Chen, Cheng an... on PithPith's reviewarXiv
Cited by 5 Pith papers
How the literature uses it

2 arXiv papers cite this in 4 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 2 sentences without a confident label

Uses the method or tool of 1 paper
Cites as support 2 papers
Cites as background 19 papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment