Pith.

Machine Learning · arXiv:2608.30730 v1 · submitted 31 Aug 2026 · 63 references

E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation

Wei Fan, Xinjie Shen, Xudong Guo, Jianhong Tu, Yang Su, Yinger Zhang, Lianghao Deng, Fengyu Wang

show 3 more authors

Baohua Dong, Yangqiu Song, Dayiheng Liu

The paper says

“Long-horizon agentic tasks go beyond chaining short tasks over more interaction turns. Their evolving dynamic environments and long-range dependencies require Large Language Models (LLMs) to continually explore, learn from experience, and adapt their policies over thousands of steps.”

more of the abstract

We introduce E-Commerce Bench, the first open-source benchmark that integrates multi-round counterpart negotiation and dynamic events into a year-long business operation. Over a 365-day year, an LLM agent concurrently runs multiple online stores, researching the market, negotiating with suppliers to source inventory, optimizing sales strategies, fulfilling orders, handling returns, and managing cash flow to maximize its end-of-year total assets. To construct a realistic merchant-side operating environment, the product and supplier data are derived from a real e-commerce platform, while a year-long calendar of promotions, natural disasters, and supply-chain shocks continually reshapes demand. For reproducibility, both sides of the market are deterministic: customer purchases and returns follow a fixed demand model, while a negotiation kernel determines supplier pricing, concessions, and decisions, with an LLM used only to verbalize them. We evaluate 18 frontier models across seven dimensions, including year-end assets, and find that no single model dominates. GPT-5.6 Sol earns the most, growing the 100,000 opening stake into 1,431,425, yet it ranks 16th of 18 on fraud avoidance and trails Fable5 in operational efficiency. Among open-weight models, Qwen3.8-Max-Preview leads with 416,252, 38% above GLM 5.2 (high), and achieves the strongest learning over the horizon, progressively bargaining down prices across repeated orders. Our code is available at https://github.com/QwenLM/E-CommerceBench.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 9 Sep 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Solid benchmark contribution undermined by a cleanly documented data leak that invalidates one of its seven evaluation axes; worth engaging but needs fixing. This line is wrong

3 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 63 references say.

The paper says, at Appendix B.5 and §4.3.2

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The fraud-avoidance axis is invalid as a measure of negotiation-based fraud detection. Appendix B.5 states that supplier_search returns an email whose local part ends in the supplier identifier and that, within every one of the 60 categories, honest suppliers have lower numeric identifiers than fraudulent ones, so an agent could screen by the numeric suffix without sending a single message.

more of objection 1

The paper acknowledges this leak but then says 'whether any evaluated model exploited it is read in §F.3, whose contact stage would show the effect'; §F.3 actually reports only outreach shares and order-conversion rates, not whether any model filtered on the identifier. Consequently BadSpend% in Table 2, §3.7, and §4.3.2 cannot be interpreted as a clean measure of fraud-detection skill: the reported ranks, including GPT-5.6 Sol's 16th and Claude Opus 4.7's 1st, may reflect ID-parsing ability rather than bargaining behavior or narrative reading. Since the 'no single model dominates' capability-profile claim explicitly relies on this axis, this is a load-bearing issue. Please randomize supplier identifiers so they are uncorrelated with the honesty label, or log and inspect the models' supplier_search usage and messages to demonstrate that the leak was not used, or restrict the fraud-avoidance claims to models shown not to have exploited the suffix.

The paper says, at §1 and §3.5.2

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The reproducibility claim is overstated. Section 3.5.2 concedes that the Layer-2 renderer is sampled at the provider's default decoding and that delivery days are drawn outside the episode seed, and §4.1 repeats that renderer wording varies between episodes. Because the narrative is explicitly described as the decisive pre-deal fraud signal, renderer variation changes what the agent observes and therefore changes the agent's policy and the resulting economic trajectory.

more of objection 2

The §1 sentence 'so a difference in outcome entirely belongs to the agent policy rather than to chance' does not follow from the stated design. The deterministic claim is correct only conditional on message content and renderer realization. Please rephrase the abstract and §1 to say that the market-side economics are deterministic functions of the agent's messages, and quantify renderer-induced outcome variance, for example by running a subset of models under a fixed renderer seed or across several renderer seeds.

The paper says, at §4.3.2 and Table 2

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The fraud-avoidance ranking is statistically fragile even if the label leak is set aside. Section 4.3.2 reports that the standard deviation of the five-episode BadSpend% exceeds its own mean for 15 of the 18 models, yet Table 2 and Figures 2 and 9 present a rank ordering as a headline capability.

more of objection 3

With five episodes per model, some adjacent fraud-avoidance ranks are within noise, and the paper provides no confidence interval on the ranks themselves. Since the central 'no single model dominates' argument depends on comparing ranks across dimensions, please add bootstrap or otherwise constructed rank intervals for the fraud axis, or increase the episode count for the models whose fraud shares have the largest relative variance.

5 minor objections

The paper says, at Abstract and §1

Pith says: minor objection 1 of 5

The word 'deterministic' is used as an unqualified property of 'both sides of the market'; please qualify it immediately with the renderer and delivery-day caveats given in §3.5.2, so that the abstract does not promise a stronger reproducibility guarantee than the paper itself establishes.

The paper says, at §3.6.2 and Table 7

Pith says: minor objection 2 of 5

The claim that supplier lookup 'ranks nothing' is contradicted by Appendix B.5's observation that the email suffix sorts honest from fraudulent suppliers within every category; the main text and Table 7 should be aligned with that admission or the leak should be fixed.

show 3 more

The paper says, at Figures 2 and 9

Pith says: minor objection 3 of 5

The radar plots and the rank strings such as '1·7·16·9·2·17·4' would be much more informative with uncertainty markers on the ranks, especially for fraud avoidance where the reported per-episode variance is large.

The paper says, at Appendix F.3

Pith says: minor objection 4 of 5

The sentence 'Every screening decision therefore ran over a narrow slice the model had picked out itself' is not an analysis of whether the email-suffix leak was used; where Appendix B.5 promises such an analysis, Appendix F.3 should actually provide it or explicitly say that the archived data cannot answer the question.

The paper says, at General

Pith says: minor objection 5 of 5

The paper should include a conflict-of-interest disclosure: the authors are affiliated with the Qwen Team and the benchmark evaluation highlights Qwen3.8-Max-Preview as the leading open-weight model; a disclosure statement and, ideally, an independent replication of the open-weight ranking would strengthen the credibility of the comparison.

What would settle it

Re-analyze the 90 archived episodes by scanning every chatbox tool call for explicit reference to the supplier ID numeric suffix (e.g., 'supplier 102') in a screening context (e.g., 'I'll avoid supplier 102' or 'contact lower IDs first'). Additionally, within each category, compute the mean supplier ID of fraudulent suppliers that each model contacted versus those it never contacted. A statistically significant difference (e.g., contacted fraudulent suppliers have lower IDs than non-contacted ones) would indicate ID-based screening. If neither test shows evidence, the concern is mitigated but not eliminated; a fully controlled experiment would be required to definitively rule out implicit use. The code and logs are released, so this check is feasible.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 9 Sep 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 24 s · 59,662 tokens read, 2,220 written
  3. Editor wrote the verdict above 14 s · 60,652 tokens read, 1,170 written
  4. Referee major revision: 3 major objections, 5 minor 57 s · 60,252 tokens read, 5,007 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

E-Commerce Bench is a genuinely serious benchmark effort. The combination of a deterministic negotiation kernel, real-data product catalog, 365-day multi-store management, and multi-dimensional evaluation is new and well-executed. The code release, detailed appendices, and honest reporting of per-model variance all show careful work. This is the kind of benchmark the agent community needs for testing long-horizon continuing tasks beyond today's episodic or stochastic setups.

The fraud-label leak in Appendix B.5 is the main problem, and it's a real one. The authors explicitly state that within every product category, honest suppliers have lower numeric identifiers than fraudulent ones, and the supplier email exposes that identifier. This means an agent can parse the email suffix to screen fraud without any negotiation or narrative reading. The claim that fraud avoidance measures 'detection from bargaining behavior' is therefore invalid for this axis, unless the authors can demonstrate that no evaluated model exploited the signal. They did not check. Since two of the paper's main selling points are the multi-dimensional capability profile and the finding that 'no single model dominates,' losing a clean interpretation of fraud avoidance weakens the headline results. The authors should either remove the leak from the public data or run a control experiment to show the leakage was unused.

The determinism claim is slightly oversold—the renderer and delivery-day draws introduce minor run-to-run variance—but that is honestly qualified in the text and doesn't affect the core findings. Five episodes per model is thin but acceptable with standard deviations reported. The authors' own model leading the open-weight tier is a conflict-of-interest flag, not a scientific flaw, since the benchmark is open-source.

This paper is for anyone working on long-horizon LLM agents. The benchmark is valuable as a reusable testbed for planning, memory, and continual learning in a rich simulated economy. The fraud-avoidance leak is load-bearing but fixable. If the authors can resolve it—either by removing the signal or by proving models did not exploit it—the paper belongs in a good venue. I would accept peer review and recommend conditional acceptance pending that revision.

The strongest objection

Appendix B.5 explicitly states that within every category, honest suppliers have lower numeric identifiers than fraudulent ones, and the supplier email exposes that identifier. This means an agent can, in principle, avoid fraudulent suppliers entirely by parsing the numeric suffix, without ever engaging in negotiation or reading narrative. The central claim of a multi-dimensional capability profile relies on the fraud-avoidance axis (BadSpend%) being a valid measure of the agent's ability to detect fraud from bargaining behavior. If agents exploited this ID leak, then the ranking on this axis is not about fraud-detection skill but about ID-parsing ability. The paper does not check whether any evaluated model used this signal. Since 18 models were evaluated, some may have inadvertently or intentionally used it. This directly undermines the interpretability of one of the seven headline dimensions, which is a key part of the paper's contribution ('Reading six dimensions beside that total turns a leaderboard into a capability profile'). Without mitigation or demonstration that no model exploited the leak, the claim that 'no single model dominates' is weakened because the fraud-avoidance scores may be artifactual.

The reader's weakest assumption is precisely the fraud-label leak. My analysis confirms this is the most load-bearing concern because it directly attacks the validity of one of the seven evaluation axes that the paper uses to support its central claim of capability profiling. The paper's transparency about the leak is commendable, but it does not demonstrate that models did not exploit it. Since the evaluation results depend on this axis (e.g., the claim that GPT-5.6 Sol ranks 16th/18 on fraud avoidance), showing that no model used the leaked signal is essential for the rankings to be interpretable. The concrete test is simple, feasible with the released logs, and would decisively confirm or refute the threat. I agree with the reader that this warrants a CONDITIONAL verdict, and my concern does not change that; it reinforces the condition.

The referee's report

If the benchmark is taken at face value, it is a substantial engineering contribution: it combines real-platform product and supplier data, deterministic market-side economics, multi-round repeated negotiation, dynamic events, and a multi-dimensional capability evaluation in one open system. The paper is unusually transparent in places: the 13-step settlement order is spelled out, seeds are fixed and reported, every metric has a formal definition in Appendix E, and per-episode trajectories and error bars are provided. The fraud-avoidance axis, however, is contaminated by a self-admitted label leak, and the determinism claim is stronger than what the paper's own caveats support; because the narrative channel is the only pre-deal fraud signal, the headline capability-profile result is not currently interpretable as a measure of fraud-detection skill. The benchmark itself is likely salvageable, and the central construction is defensible, but the evaluation claims need repair before the paper's main conclusions can be accepted.

Pith's reading of the paper (a machine's summary, not the authors')

E-Commerce Bench is a new open-source benchmark that simulates a year-long online retail business for LLM agents, with deterministic demand and a deterministic negotiation kernel to ensure reproducibility. The paper argues that this benchmark reveals capability profiles that total assets alone miss: the top earner (GPT-5.6 Sol) grows capital 14-fold but is nearly worst at fraud avoidance, while Fable5 earns more per tool call. Among open-weight models, Qwen3.8-Max-Preview leads and uniquely shows learning over repeated purchases. If correct, the benchmark provides a reproducible, multidimensional testbed revealing that current frontier models are strong in patches but coordinated nowhere.

What carries the argument. The central mechanism is the deterministic negotiation kernel combined with a fixed demand model. The negotiation kernel sets every supplier price, concession, and accept/reject decision via explicit bargaining policies, while an LLM only renders dialogue. This eliminates evaluation noise from stochastic supplier responses. The demand model uses a multi-factor formula with price elasticity, seasonality, promotions, events, and reputation. Together they make the market deterministic except for wording variance, so outcome differences belong entirely to agent policy.

Core claim. The paper introduces E-Commerce Bench, the first open-source benchmark that integrates deterministic counterpart negotiation and dynamic events into a year-long business operation. Eighteen models are evaluated on seven dimensions: end-of-year total assets, negotiation quality, fraud avoidance, cash flow and solvency, operational efficiency, operations execution, and learning over the horizon. The central finding is that no single model dominates: GPT-5.6 Sol earns the most assets (¥1,431,425) yet ranks 16th of 18 on fraud avoidance and trails Fable5 in efficiency. Claude Opus 4.7 leads bargaining and fraud avoidance but finishes mid-pack on profit. Among open-weight models, Qwen3.8-Max-Preview leads with ¥416,252 and is the only model showing significant learning over repeated negotiations. The paper claims that deterministic kernel control eliminates evaluation noise and that multidimensional diagnostics are indispensable for catching diverse failure modes.

The premise it rests on. The fraud-avoidance axis assume that BadSpend% measures fraud-detection skill, but the supplier email leaks the fraud label via numeric identifiers, so unless no model exploited this signal, the metric is not clean.

If the paper is right (Pith's inferences, not the authors')

LLM agent evaluations must move beyond single-score leaderboards to multidimensional capability profiles.

Deterministic kernel control can provide reproducible evaluation for multi-party interactions.

High total-assets growth can mask severe weaknesses in fraud detection and operational discipline.

Current frontier models show little to no learning over long horizons, suggesting a fundamental deficiency in experiential learning.

The benchmark enables systematic study of which agent design choices drive which capabilities.

The fraud-avoidance measure (BadSpend%) may be confounded by a subtle signal: in the released supplier data, honest suppliers have lower numeric identifiers than fraudulent ones within each category, so an agent screening by email suffix could achieve low BadSpend% without genuine fraud-detection skill. The paper does not demonstrate that no evaluated model used this signal.

The claim of full reproducibility is slightly overstated: the LLM renderer's wording varies between runs, and delivery-day draws are outside the episode seed, adding noise to the fraud dimension.

The learning metric (AnchorRatio) measures price discipline but does not separate three distinct mechanisms: context eviction, memory underuse, and lack of reasoning over past prices. A targeted ablation study could disentangle them.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Long-horizon agentic tasks go beyond chaining short tasks over more interaction turns. Their evolving dynamic environments and long-range dependencies require Large Language Models (LLMs) to continually explore, learn from experience, and adapt their policies over thousands of steps. We introduce E-Commerce Bench, the first open-source benchmark that integrates multi-round counterpart negotiation and dynamic events into a year-long business operation. Over a 365-day year, an LLM agent concurrently runs multiple online stores, researching the market, negotiating with suppliers to source inventory, optimizing sales strategies, fulfilling orders, handling returns, and managing cash flow to maximize its end-of-year total assets. To construct a realistic merchant-side operating environment, the product and supplier data are derived from a real e-commerce platform, while a year-long calendar of promotions, natural disasters, and supply-chain shocks continually reshapes demand. For reproducibility, both sides of the market are deterministic: customer purchases and returns follow a fixed demand model, while a negotiation kernel determines supplier pricing, concessions, and decisions, with an LLM used only to verbalize them. We evaluate 18 frontier models across seven dimensions, including year-end assets, and find that no single model dominates. GPT-5.6 Sol earns the most, growing the 100,000 opening stake into 1,431,425, yet it ranks 16th of 18 on fraud avoidance and trails Fable5 in operational efficiency. Among open-weight models, Qwen3.8-Max-Preview leads with 416,252, 38% above GLM 5.2 (high), and achieves the strongest learning over the horizon, progressively bargaining down prices across repeated orders. Our code is available at https://github.com/QwenLM/E-CommerceBench.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 1.00

“We evaluate 18 frontier models across seven dimensions, including year-end assets, and find that no single model dominates.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.95 · states limitation 0.52 · human participants 0.03 · controlled comparison 0.12 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.60 · mentions code or data availability 0.98

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Long-horizon agentic tasks go beyond chaining short tasks over more interaction turns. background 0.99 0.07 0.03 asserted 1.00
Their evolving dynamic environments and long-range dependencies require Large Language Models (LLMs) to continually explore, learn from experience, and adapt their policies over thousands of steps. background 0.99 0.06 0.03 asserted 1.00
We introduce E-Commerce Bench, the first open-source benchmark that integrates multi-round counterpart negotiation and dynamic events into a year-long business operation. contribution 1.00 0.95 0.07 asserted 1.00
Over a 365-day year, an LLM agent concurrently runs multiple online stores, researching the market, negotiating with suppliers to source inventory, optimizing sales strategies, fulfilling orders, handling returns, and managing cash flow to maximize its end-of-year total assets. method 0.58 0.63 0.05 asserted 0.99
To construct a realistic merchant-side operating environment, the product and supplier data are derived from a real e-commerce platform, while a year-long calendar of promotions, natural disasters, and supply-chain shocks continually reshapes demand. method 0.99 0.88 0.02 asserted 1.00
For reproducibility, both sides of the market are deterministic: customer purchases and returns follow a fixed demand model, while a negotiation kernel determines supplier pricing, concessions, and decisions, with an LLM used only to verbalize them. method 1.00 0.84 0.03 asserted 1.00
We evaluate 18 frontier models across seven dimensions, including year-end assets, and find that no single model dominates. result 0.95 0.97 0.48 asserted 1.00
GPT-5.6 Sol earns the most, growing the 100,000 opening stake into 1,431,425, yet it ranks 16th of 18 on fraud avoidance and trails Fable5 in operational efficiency. result 1.00 0.94 0.14 asserted 1.00
Among open-weight models, Qwen3.8-Max-Preview leads with 416,252, 38% above GLM 5.2 (high), and achieves the strongest learning over the horizon, progressively bargaining down prices across repeated orders. result 1.00 0.95 0.11 asserted 1.00
Our code is available at https://github.com/QwenLM/E-CommerceBench. contribution 0.84 0.69 0.02 asserted 0.97

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.53

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,813 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (30)
Figure 1 of the paper
Figure 1. Mean end-of-year total assets, eighteen models, five 365-day episodes each. Bars start at the
Figure 2 of the paper
Figure 2. Seven evaluation axes for the strongest model of each vendor family by mean total assets, the
Figure 3 of the paper
Figure 3. Overview of E-Commerce Bench. The agent works through four layers. An
Figure 4 of the paper
Figure 4. The agent loop layer. One model call produces a batch of tool calls; the batch runs in order,
Figure 5 of the paper
Figure 5. How eviction works. A group is what the editor removes as a unit: one model reply together
Figure 6 of the paper
Figure 6. Three-account deferred settlement. (a) Costs leave the bank the day they are incurred, while
Figure 7 of the paper
Figure 7. The demand model. (a) The four price-elasticity families at their median parameter, solid
Figure 8 of the paper
Figure 8. Negotiation at two timescales, from real episodes, each panel on its own price axis and the
Figure 9 of the paper
Figure 9. Capability profiles for six of the 18 models. The primary score and the six dimensions of this
Figure 10 of the paper
Figure 10. (a) Profit per tool call for the 18 models, best first, mean total assets less the
Figure 11 of the paper
Figure 11. One bankruptcy episode day by day, Qwen3.5-Plus episode 0. Total assets and bank balance
Figure 12 of the paper
Figure 12. How often the context editor fired, over the 90 evaluated episodes. Each event is one eviction
Figure 13 of the paper
Figure 13. The released data. (a) Products per category. (b) Reference prices. (c) Natural return rate by
Figure 14 of the paper
Figure 14. The monthly seasonality multiplier for all 12 store types, rows grouped by difficulty tier. The
Figure 15 of the paper
Figure 15. The 2026 calendar. Eight promotions the agent can join by choosing a discount, and ten events
Figure 16 of the paper
Figure 16. The thirteen settlement steps in the order the simulation runs them at each 08:00 crossing,
Figure 17 of the paper
Figure 17. How reputation moves. (a) The goodwill term against units shipped. (b) The penalty term
Figure 18 of the paper
Figure 18. Where a unit’s money goes. (a) Each component as a share of retail price, for a median-priced
Figure 19 of the paper
Figure 19. The four decision functions of the Negotiation Kernel, swept on one supplier and SKU. (a)
Figure 20 of the paper
Figure 20. One negotiation session, abridged, from GPT-5.6 Sol. Agent wording, prices and rounds are
Figure 21 of the paper
Figure 21. Total assets over the year for eight models spanning the ranking. The solid line is the mean
Figure 22 of the paper
Figure 22. Total assets over the year for all 18 models, ordered by their mean at the end of the year. Each
Figure 23 of the paper
Figure 23. How much of each bargaining range models capture against each kind of supplier, over all
Figure 24 of the paper
Figure 24. Honest procurement money split over the six counterpart behavior templates, which run easiest
Figure 25 of the paper
Figure 25. (a) The share of procurement spend that reaches fraudulent suppliers, cleanest model first,
Figure 26 of the paper
Figure 26. How fraudulent suppliers get through, 18 models and 5 episodes each, counted as distinct
Figure 27 of the paper
Figure 27. (a) The deepest fall in total assets, as a share of that episode’s peak, against end-of-year total
Figure 28 of the paper
Figure 28. The same capability profiles for all 18 models, with a dashed polygon at the median. Axes
Figure 29 of the paper
Figure 29. How the ranking changes from one dimension to the next, all 18 models, rank 1 at the top of
Figure 30 of the paper
Figure 30. How often each failure rule fires. Rows are the rules of Table

Figures are the authors', reproduced from the paper.

References (63)
  1. [1] Sum the counts of every message not already marked cleared. Below that ceiling the pass emits the usage string and frees nothing on Pith
  2. [2] Scan for groups from the third message onward, leaving the system message and the first user turn permanently out of reach. A group opens at an assistant message carrying tool calls and closes after the last consecutive tool response behind it on Pith
  3. [3] Set aside the two most recent groups not yet cleared. With two or fewer active groups the pass clears nothing at all, so a history made of a few very large groups can sit above the ceiling indefinitely on Pith
  4. [4] Take the release target as the larger of that floor and the current overshoot past the ceiling on Pith
  5. [5] Walk the clearable groups oldest first, marking each cleared and accumulating its count, and stop at the first group that meets the target on Pith
  6. [6] Rebuild the request from the surviving messages, skipping cleared entries entirely (§3.2.2). Whether a pass ever fires depends on how a model spends the window, and Figure 12 counts the passes of each of the 90 episodes. Two kinds of message can never be evicted, an assistant ... on Pith
  7. [7] Kimi K3 5-run mean 259k on Pith
  8. [8] Claude Opus 4.7 5-run mean 258k2 of 5 bankrupt on Pith
  9. [9] Return-rate wording understates the numeric band.Twenty of the 60 categories carry a sentence reading low or very low against a numeric band reaching 20% to 35%, marked in Table 4. The sentence is what market_search shows, so a merchant who prices off the wording underestimate... on Pith
  10. [10] The supplier email leaks the fraud label.supplier_search returns an email whose local part ends in the supplier identifier, and inside every one of the 60 categories the honest suppliers hold lower identifiers than the fraudulent ones, so an agent sorting on that numeric suffi... on Pith
  11. [11] Terminal outcome only.Aggregation runs over the sessions whose recorded outcome is agreement or disagreement, so a record still open when the episode stops enters no metric on Pith
  12. [12] Settled agreements only.An agreement is filed after the purchase order has charged the bank account, so an agreed price whose order failed leaves no session behind (Appendix D.4) on Pith
  13. [13] Honest counterpart only, for the value-extraction metrics.ZG keeps the sessions whose supplier is one of the 424 honest names on the 576-name supplier list, andZB holds the fraudulent remainderB. Symbols follow §3.5.1. Sessionz closes at an agreed unit pricepz, or atfz =⊥ when... on Pith
  14. [14] Nothing in the economy charges for a call or a token, so the axis prices a cost the primary score cannot see. Only the 600-minute working day constrains the agent from inside the simulation on Pith
  15. [15] Waiting on the clock spends no simulated minutes yet still costs one call, so a call count is not a proxy for how much of the working day a model used on Pith
  16. [16] A turn that requests no tool still counts as a turn, so a model that deliberates without acting pushes ¯n/¯Tbelow one, and three such turns in a row end the episode on Pith
  17. [17] A truncated episode contributes fewer calls, fewer turns, fewer eviction passes and fewer memory actions to every mean it enters, which is why GPT-5.5 and Qwen3.5-Plus carry the widest call spreads in Table 14. E.5 Operations Execution Selling, shipping and opening stores leav... on Pith
  18. [18] The axis is an expectation accrued when a parcel leaves rather than a count of units that came back, and it turns negative where a model priced under the reference, as it does in 26 of the 90 episodes on Pith
  19. [19] Every rate divides by a count floored at one, so an episode that shipped nothing records0rather than nothing. Two GPT-5.5 episodes enter its return means that way on Pith
  20. [20] On-time share is one minus the cancelled share of orders sold, so an episode with no sales scores0and pulls a five-episode mean far below the years that traded on Pith
show 43 more
  1. [21] Refund loss and return freight are gross¥totals that grow with units sold, which makes them readable against a model’s own sales and not across models of different size. 5.α j is a ratio over cumulative deliveries pooled by SKU, so the defective channel names the suppliers a m... on Pith
  2. [22] Fraudulent pairs fall away, since a pre-deal fraudulent floor is inflated by construction and descent toward it is not the same quantity (§3.5.4) on Pith
  3. [23] An agreement with no recorded conclusion day is dropped on Pith
  4. [24] A pair whose range satisfies∆P≤0goes with it on Pith
  5. [25] A pair transacted once contributes nothing, and an episode holding no repeat pair returns null and drops out of the model mean. Repeat purchases of the same item from the same supplier are common enough to score. The filters leave 2,230 such pairs over the 90 episodes, and Tab... on Pith
  6. [26] GPT-5.6 Sol 5-run mean 805k on Pith
  7. [27] Fable5 5-run mean 702k2 of 5 bankrupt on Pith
  8. [28] GPT-5.5 10k 100k 1M assets (¥, log) 5-run mean 498k on Pith
  9. [29] Claude Opus 4.8 5-run mean 416k on Pith
  10. [30] Qwen3.8-Max-Preview 5-run mean 301k on Pith
  11. [31] GLM 5.2 (high) 10k 100k 1M assets (¥, log) 5-run mean 265k on Pith
  12. [34] Claude Opus 4.6 10k 100k 1M assets (¥, log) 5-run mean 226k on Pith
  13. [35] GLM 5.1 5-run mean 190k on Pith
  14. [36] DeepSeek-V4-Pro-Preview 5-run mean 190k on Pith
  15. [37] Gemini 3.5 Flash 10k 100k 1M assets (¥, log) 5-run mean 165k on Pith
  16. [38] Qwen3.7-Max 5-run mean 130k2 of 5 bankrupt on Pith
  17. [39] Gemini 3.1 Pro 5-run mean 115k on Pith
  18. [40] GLM 5.2 (max) 1 120 240 365 simulated day 10k 100k 1M assets (¥, log) 5-run mean 70k on Pith
  19. [41] Kimi K2.6 1 120 240 365 simulated day 5-run mean 47k on Pith
  20. [42] Qwen3.6-Plus 1 120 240 365 simulated day 5-run mean 1k4 of 5 bankrupt on Pith
  21. [43] Qwen3.5-Plus Figure 22: Total assets over the year for all 18 models, ordered by their mean at the end of the year. Each thin line is one episode and the black line their mean while they are still running. The dashed red line marks the¥100,000 stake and a red cross marks a ban... on Pith
  22. [44] DeepSeek-V4-Pro-Preview on Pith
  23. [45] Qwen3.5-Plus field, deal-weighted 0.721 369 0.606 101 0.645 238 0.639 199 0.648 226 0.590 58 0.662 1191 0.618 221 0.732 301 0.858 136 0.758 144 0.732 126 0.728 152 0.796 12 0.756 871 0.678 15 0.654 289 0.683 133 0.676 370 0.673 145 0.669 148 0.693 109 0.672 1194 0.686 245 0.61... on Pith
  24. [46] GPT-5.6 Sol rank 1·7·16·9·2·17·4 0.5 1.0 Profit Negotiation Fraud avoidance SolvencyEfficiency Execution Learning on Pith
  25. [47] Fable5 rank 2·3·4·3·1·4·17 0.5 1.0 Profit Negotiation Fraud avoidance SolvencyEfficiency Execution Learning on Pith
  26. [48] GPT-5.5 rank 3·5·14·16·4·10·3 0.5 1.0 Profit Negotiation Fraud avoidance SolvencyEfficiency Execution Learning on Pith
  27. [49] Claude Opus 4.8 rank 4·9·5·2·3·3·7 0.5 1.0 Profit Negotiation Fraud avoidance SolvencyEfficiency Execution Learning on Pith
  28. [50] Qwen3.8-Max-Preview rank 5·4·7·7·5·6·1 0.5 1.0 Profit Negotiation Fraud avoidance SolvencyEfficiency Execution Learning on Pith
  29. [51] GLM 5.2 (high) rank 6·6·2·1·7·8·13 0.5 1.0 Profit Negotiation Fraud avoidance SolvencyEfficiency Execution Learning on Pith
  30. [52] Kimi K3 rank 7·13·6·8·9·13·14 0.5 1.0 Profit Negotiation Fraud avoidance SolvencyEfficiency Execution Learning on Pith
  31. [53] Claude Opus 4.7 rank 8·1·1·5·6·12·11 0.5 1.0 Profit Negotiation Fraud avoidance SolvencyEfficiency Execution Learning on Pith
  32. [54] Claude Opus 4.6 rank 9·12·13·15·8·9·6 0.5 1.0 Profit Negotiation Fraud avoidance SolvencyEfficiency Execution Learning on Pith
  33. [55] GLM 5.1 rank 10·8·8·6·10·2·16 0.5 1.0 Profit Negotiation Fraud avoidance SolvencyEfficiency Execution Learning on Pith
  34. [56] DeepSeek-V4-Pro-Preview rank 11·11·9·4·11·11·5 0.5 1.0 Profit Negotiation Fraud avoidance SolvencyEfficiency Execution Learning on Pith
  35. [57] Gemini 3.5 Flash rank 12·2·3·13·13·5·2 0.5 1.0 Profit Negotiation Fraud avoidance SolvencyEfficiency Execution Learning on Pith
  36. [58] Qwen3.7-Max rank 13·17·10·12·12·1·12 0.5 1.0 Profit Negotiation Fraud avoidance SolvencyEfficiency Execution Learning on Pith
  37. [59] Gemini 3.1 Pro rank 14·10·12·17·14·14·10 0.5 1.0 Profit Negotiation Fraud avoidance SolvencyEfficiency Execution Learning on Pith
  38. [60] GLM 5.2 (max) rank 15·14·17·10·15·15·9 0.5 1.0 Profit Negotiation Fraud avoidance SolvencyEfficiency Execution Learning on Pith
  39. [61] Kimi K2.6 rank 16·18·15·11·16·7·15 0.5 1.0 Profit Negotiation Fraud avoidance SolvencyEfficiency Execution Learning on Pith
  40. [62] Qwen3.6-Plus rank 17·15·11·14·17·16·8 0.5 1.0 Profit Negotiation Fraud avoidance SolvencyEfficiency Execution Learning on Pith
  41. [63] Qwen3.5-Plus rank 18·16·18·18·18·18·18 0.5 1.0 axes min-max normalized over the 18 model means 18-model median model mean over 5 runs ±1 standard deviation over 5 runs Figure 28: The same capability profiles for all 18 models, with a dashed polygon at the median. Axes clockwis... on Pith
  42. [2025] Association for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10.18653/v1/2025.findi ngs-emnlp.240. URLhttps://aclanthology.org/2025.findings-emnlp.240/. Kian Siong Tey, Michael Schaerer, Nikhil Madan, and Roderick I. Swaab. The impact of concession patterns onnegoti... on PithPith's reviewarXiv
  43. [2026] URLhttps://arxiv.org/abs/2607.05155. Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, Cozmin Ududec, ... on PithPith's reviewarXiv
Cited by 6 Pith papers
How the literature uses it

What this paper does with the arXiv papers it cites

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 18 sentences without a confident label

Contests 1 paper
Builds on 1 paper
Uses the method or tool of 2 papers
Compares with 1 paper
  • TERMS-Bench [ref] sits outside the business-operation line, diagnosing bilateral price bargaining against a rule-based counterpart over sampled regimes rather than data-grounded cost floors, so it supplies a scripted opponent but no economy for the resulting deals to act on.

    TERMS-Bench: Diagnosing LLM Negotiation Agents Beyond Deal Rate · Continuing Long-Horizon Agentic Tasks · 0.94

Cites as background 6 papers
  • The shape recurs across trading a live portfolio for cumulative return [ref] and self-evolving agents that grow a skill library [ref] or rewrite their own code [ref] .

    2510.02209 · Continuing Long-Horizon Agentic Tasks · 0.81

  • The shape recurs across trading a live portfolio for cumulative return [ref] and self-evolving agents that grow a skill library [ref] or rewrite their own code [ref] .

    Voyager: An Open-Ended Embodied Agent with Large Language Models · Continuing Long-Horizon Agentic Tasks · 0.81

  • The shape recurs across trading a live portfolio for cumulative return [ref] and self-evolving agents that grow a skill library [ref] or rewrite their own code [ref] .

    Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents · Continuing Long-Horizon Agentic Tasks · 0.81

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment