{"id":"14935be0-682b-4b4d-ae9f-dc5828b79d9e","arxiv_id":"2608.00102","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLM merchant agents in a new dynamic auction benchmark capture at most 32% of hindsight-optimal profit; profit tracks margin per win more than win rate, and fast pre-shock learners adapt poorly to preference shocks.","lead":"Visa Research built Bazaar, a simulated market where LLM agents choose product configurations and prices for customers with hidden preferences that can shift mid-game. They found that today's frontier models earn less than a third of the theoretical best profit, and that winning more auctions does not mean earning more money.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 4's hindsight oracle uses continuous prices while actual bids are integer-constrained (Appendix P); the '<1/3 efficiency' headline may be inflated by the oracle denominator.","rationale":"The most load-bearing condition for the headline claim is that regret efficiency η has a correctly computed denominator. Eq. 4 uses a continuous price optimum, while the implemented agents are required to submit integer prices (Appendix P). A per-auction overstatement of <$1 in the oracle is enough to move Opus 4.6's η from 0.321 to above 1/3 after ~1,900 customer-rounds, which would falsify the abstract's 'less than a third' wording. This is not a consensus disagreement; it is an internal consistency issue between the metric and the action space. The reader's weakest assumption (pre-shock window) affects learning-speed/recovery rankings, which are secondary to the headroom claim. The paper's robustness check (§4.4, fixed-competitor oracle) is good but does not fix the integer-price inconsistency because it only replaces competitor utility values. A direct recomputation of the oracle under integer prices is inexpensive and would settle the question. Even if the corrected η stays below 1/3, the manuscript needs to report integer-consistent oracle values and adjust the text; if it crosses 1/3, the central numerical claim must be revised. Either way the qualitative 'substantial headroom' conclusion likely survives, so the verdict remains conditional rather than accept/reject.","tokens_in":25227,"tokens_out":11144,"duration_ms":132202,"concrete_test":"Recompute all rows of Table 5/Table 6 with an integer-constrained oracle: for each customer-round set π⋆_{i,t} = max_{x, integer p : c_f(x) ≤ p < V_{i,t}(x) − q_{i,t}} (p − c_f(x)), and recompute η, Under, and Loss. Compare the new top η to 1/3. If Opus 4.6 (adaptive/xhigh) η exceeds 1/3, the abstract's 'less than a third' claim is an artifact; if it remains below, the claim survives numerically but should be reported with integer-consistent oracle values.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Prices in the implemented benchmark are whole numbers (Appendix P: 'price': <integer >= minimum cost; §3.3 examples like (3,2,1) @ $84). But the oracle in Eq. (4), π⋆_{i,t} = max_x [V_{i,t}(x) − q_{i,t} − c_f(x)]_+, treats price as continuous. For a winnable customer-round, the continuous optimum exceeds the best integer-feasible profit by the fractional part of V−q (or by 1 when V−q is an integer). With 24 customers and ~80 rounds, ~1,900 customer-rounds, this can add on the order of several hundred dollars to the oracle denominator. For Opus 4.6 (adaptive/xhigh), realized profit is $2,976 and reported oracle ≈ $9,282; a correction of only ~$800 drops the oracle to ~$8,480 and raises η from 0.321 to ~0.35 — above the 'less than a third' threshold. The exact correction depends on the distribution of fractional parts, but the claim as stated is not robust to the benchmark's own integer-price constraint. The regret decomposition (Under/Loss) is similarly affected because underprice regret should be measured against integer-optimal prices. This is a metric-construction issue, not a protocol choice, and it directly targets the paper's headline number.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Bazaar, a repeated sealed-bid multi-attribute auction benchmark in which an LLM merchant competes with three adaptive specialist bots for 24 customers with hidden additive preferences. A staggered preference shock tests adaptation. The environment uses closed-form utilities and costs, so surplus and hindsight regret can be computed exactly. The authors evaluate eleven frontier LLMs and their variants, reporting that total profit spans an 11x range, correlates strongly with margin per win (r=0.99) rather than win rate (r=0.88), that thinking effort shifts models from Loser to Underpricer failure archetypes, and that strong pre-shock learners are typically weak post-shock adapters. The best reported efficiency is η=0.321, i.e., less than a third of hindsight-optimal profit, leaving substantial headroom.","tokens_in":25540,"tokens_out":7125,"duration_ms":83899,"significance":"Bazaar is a valuable evaluation instrument: it combines learning, pricing, and belief revision in a dynamic competitive setting, uses exact oracle-based diagnostics rather than learned proxies, includes Thompson Sampling and EXP4 baselines, reports token/API costs, and states that prompts and logs are released. The margin-per-win vs. win-rate dissociation and the loss/underprice regret decomposition are genuinely useful diagnostics. The main central claims, however, rest on the exactness of the oracle and on the chosen pre-shock window. The integer-price issue and the empirical window selection are fixable and do not invalidate the benchmark design, but they affect the headline efficiency number and the adaptation findings.","major_comments":[{"comment":"The oracle in Eq. (4) treats price as continuous, but the implemented game constrains bids to integer prices (Appendix P: 'price': <integer >= minimum cost>; §3.3 examples). For a winnable customer-round, the continuous oracle profit is V_i,t(x)-q_i,t-c_f(x), while the best integer-feasible profit is floor(V_i,t(x)-q_i,t)-c_f(x) (or one less if ties are not guaranteed wins). The oracle is therefore inflated by the fractional part of V-q on every winnable round. Across 24 customers and ~80 rounds this can be hundreds of dollars. For Opus 4.6 (adaptive/xhigh), reported profit $2,976 and oracle ~$9,282 give η=0.321; a correction of a few hundred dollars moves η above 1/3, so the headline 'less than a third of hindsight-optimal profit' is not robust. The underprice-regret component in §4.4 is likewise measured against a continuous optimum. Please recompute all η values, tier boundaries, and","section":"§2.5, Eq. (4); Appendix P"},{"comment":"The 30-round pre-shock window is stated to have been 'chosen empirically across our model sweep.' Learning speed is measured as the rise in win rate to the R26–30 peak, and shock recovery is measured relative to that peak. If some models had not saturated by round 30, their post-shock trajectories conflate continued pre-shock learning with adaptation; selecting the window after observing model trajectories also risks overfitting the reversal. The strong-learner/weak-adapter reversal is a load-bearing contribution. Please provide a principled saturation criterion and sensitivity analyses over window lengths (e.g., 20, 30, 40) and shock timings, and report whether the reversal and recovery rankings persist.","section":"§3.5, §4.2"},{"comment":"Table 3 places GPT-5.5 xhigh in Tier 1 with a footnote '‡3 seeds rather than 10', and Table 6 again marks this row as 3-seed. This is inconsistent with the text's claim that all models are averaged over ten seeds (with ten seeds fixing bot randomness and shock schedules). The permutation-test tier boundaries in §4.1 are computed on seed-level outcomes; a 3-seed row cannot be placed in a tier with the same confidence. The same row also supports the efficiency claim 0.307 and the statement that GPT-5.5 xhigh 'clears GPT-5.3.' Please either run that cell to 10 seeds or explicitly mark the row as preliminary and exclude it from tier and efficiency comparisons. Also state the tie-breaking rule for equal-utility offers; Eq. (4) appears to assume that a tie is a win.","section":"Table 3; Table 6; §3.5"}],"minor_comments":[{"comment":"Typo: 'met:ics' should be 'metrics.'","section":"§3.7"},{"comment":"The bid-stage prompt states that ties reveal nothing because 'the winner was random.' If tie-breaking is random, the oracle in Eq. (4) should account for strict preference rather than silently assuming a tie wins, or the tie rule should be defined and enforced in the environment.","section":"§2.5 / Appendix P"},{"comment":"The main text says estimated API cost varies 14× across models and that Gemini 3.1 Pro is the most cost-effective; Appendix J says cost varies ~50×, with GPT-5.3 Chat the cheapest at $26/seed. Please reconcile these numbers and clarify which subset each figure refers to.","section":"§4.5 vs. Appendix J"},{"comment":"The EXP4 update uses K both for the number of arms and for the attribute count in the main text; use distinct symbols or clarify the notation.","section":"Appendix O"},{"comment":"The anchor probes use 3 seeds and 30 rounds. The caveat says they are confirmatory; this is acceptable, but the main text should not cite them as evidence beyond their stated power.","section":"Appendix L"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and diagnostic framework are solid, and the integer-constrained oracle correction is a routine but necessary re-computation. The empirical pre-shock window choice needs sensitivity analysis before the adaptation findings can be accepted. I do not see grounds for rejection. The 3-seed GPT-5.5 xhigh row should be brought to the 10-seed standard or excluded from comparative claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my take. Bazaar is a genuinely useful benchmark for LLM pricing: repeated sealed-bid multi-attribute auctions with hidden customer values, adaptive bot competitors, unannounced preference shocks, and closed-form utilities that make the regret decomposition exact. That combination is new, and it produces interesting, credible findings: margin per win separates models more than win rate, thinking budget shifts failure type from loser to underpricer, and fast pre-shock learners are often slow adapters. The behavioral traces (lock-in ledgers, restricted search, pruning) are persuasive qualitative evidence. The bandit baselines are a sensible floor, and the benchmark differentiates models. The main soft spot is the oracle. The implementation requires the LLM's price to be an integer (Appendix P), but the hindsight oracle in Eq. 4 optimizes over continuous price, so the denominator is not achievable. I checked the second-pass concern about integer-constrained prices: it holds. Over roughly 1,900 customer-rounds the fractional parts add up - likely a few hundred dollars. For the best agent, efficiency 0.321 would move to roughly 0.35 if you redo the oracle with integer-feasible prices, which flips the less than a third headline to more than a third. The rankings and qualitative findings survive, but that specific number is not robust to the benchmark's own constraint. Fix this by either allowing continuous prices in the environment or computing the integer-constrained oracle. Second, the r=0.99 profit-margin correlation is partly mechanical (profit = wins times margin per win), so it should be described as a near-decomposition, not a deep behavioral insight. The 30-round pre-shock window was chosen empirically across the model sweep; that is acceptable but should be varied or justified if the adaptation-speed claims are to be firm. Two rows use only 3 seeds instead of 10, and the reproducibility artifacts are announced but not actually locatable - no repo URL or appendix pointer. None of this undercuts the central value of the benchmark. It deserves a serious referee, but the revision needs to fix the oracle, harmonize seed counts, and make the artifacts findable before I would rely on the exact efficiency numbers. Send it out for review with those requests.","headline":"A useful new benchmark for LLM pricing, but the headline oracle-efficiency number is inflated by an integer-price mismatch; the qualitative findings hold.","tokens_in":655,"tokens_out":1569,"would_cite":true,"duration_ms":65234,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces Bazaar, a benchmark showing that frontier LLM merchants in a dynamic auction market capture less than a third of hindsight-optimal profit, with profit driven by margin discipline rather than win rate.","keywords":["agentic commerce","LLM pricing","multi-attribute auction","regret decomposition","demand shock","margin discipline","benchmark","sealed-bid auction"],"falsifier":"Re-run Bazaar with the pre-shock window extended from 30 to 60 rounds, keeping the same shock schedule; if the fastest pre-shock learners still show the largest post-shock degradation after full saturation, the strong-learner/weak-adapter claim is supported, and if they recover as well as the fastest adapter, the reversal was an artifact of the window.","tokens_in":25074,"feed_emoji":"🛒","tokens_out":4462,"duration_ms":48722,"temperature":0.7,"pith_summary":"This paper asks whether LLM agents can act as merchants—learning hidden customer preferences, setting prices, and revising beliefs when demand shifts—in a repeated competitive auction. It introduces Bazaar, a sealed-bid multi-attribute auction benchmark with closed-form utilities, and evaluates 11 frontier LLMs. The central finding is that even the best agent captures less than a third of the profit a hindsight-optimal seller would earn. Profit is almost perfectly explained by margin per win, not win rate; more thinking effort helps models win more but can leave money on the table. A demand-shock experiment reveals that the fastest pre-shock learners are often the slowest to revise, a failure mode static benchmarks cannot see.","feed_headline":"Frontier LLM merchants capture under one-third of optimal profit","feed_subtitle":"Bazaar benchmark: profit tracks margin per win, not win rate; thinking effort shifts failure modes.","key_machinery":"The central object is Bazaar's repeated sealed-bid multi-attribute auction: each round, an LLM merchant submits a configuration over three attributes and a price to each of 24 customers with hidden additive utilities; the customer picks the offer with highest utility. Three rule-based specialist bots adapt their prices, and 12 customers undergo an unannounced pairwise swap of their value curves mid-game. Because utilities and costs are closed-form, the paper can compute a hindsight-oracle profit and decompose regret into loss regret (auctions the oracle would have won) and underprice regret (margin left on the table in won auctions). The margin-per-win statistic is the diagnostic that carrie","core_discovery":"The paper's central claim is that current frontier LLMs, when placed in a dynamic market loop, can learn to win customers but not to price profitably: total profit across models spans an 11x range, and the strongest agent reaches only 0.321 regret efficiency, about a third of the hindsight-optimal profit. Profit correlates with margin per win at r=0.99, versus r=0.88 with win rate, so the decisive skill is charging enough, not merely winning. Increasing inference effort moves models along a two-dimensional failure surface: it reduces lost-auction regret but increases under-pricing regret. When preferences shift without warning, models that learned fastest before the shock tend to adapt slowe","pith_inferences":["If margin discipline is the binding constraint, evaluation and monitoring of deployed pricing agents should track surplus extraction (margin per win relative to a ceiling), not just win rate or revenue.","The strong-learner/weak-adapter pattern suggests that confident exploitation strategies may trade away adaptivity; a testable extension is to add explicit hypothesis-revision prompts after consecutive losses and measure recovery.","The benchmark pits one LLM against rule-based bots; running full LLM-vs-LLM competition could compress or amplify the observed gaps, and would test whether the bots' asymmetric price adjustment makes the environment easier or harder than live markets.","Because customer preferences are synthetic archetypes, a natural follow-up is to ground Bazaar in real purchase or preference data; the regret-decomposition machinery transfers directly, but the absolute profit gaps may change."],"forward_implications":["If the central finding holds, inference-time compute is a first-order determinant of agentic-commerce performance: GPT-5.4 earns 7.3x more profit at high effort than at none.","Rankings are objective-dependent: the model that wins the most customers (77.9% win rate) is not the profit leader, so reporting coverage alone can mislead.","Fast pre-shock learning does not imply fast post-shock adaptation; agentic-commerce benchmarks should include unannounced preference shifts to surface this failure mode.","Classical bandits set a non-trivial floor: per-customer Thompson Sampling beats six of eleven base LLMs on profit.","Newer model versions are not reliably better: a newer model in three of four families regresses against an earlier version at default settings, and thinking effort can reverse those gaps."],"fun_headline_variants":["LLM agents win customers but lose profit, capture 32% of optimal","Bazaar benchmark: LLM pricing leaves two-thirds of potential profit","Profit gap: LLM merchants earn 0.32 of optimal despite winning bids","Fast learners adapt slowest: LLM agents struggle with demand shifts","LLM auctions: margin per win drives profit, not win rate"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing assumption is that 30 rounds give every model enough time to saturate before the preference shock; if some models are still learning when the shock hits, the measured learning-speed and recovery rankings could be artifacts of where the shock is placed.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents win customers but lose profit, capture 32% of optimal","Bazaar benchmark: LLM pricing leaves two-thirds of potential profit","Profit gap: LLM merchants earn 0.32 of optimal despite winning bids","Fast learners adapt slowest: LLM agents struggle with demand shifts","LLM auctions: margin per win drives profit, not win rate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000254,"raw_usage":{"total_tokens":1403,"prompt_tokens":743,"completion_tokens":660,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":564}},"tokens_in":487,"tokens_out":660,"duration_ms":6936,"temperature":1.0,"reasoning_tokens":564,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T01:16:47.542819+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run Bazaar with the pre-shock window extended from 30 to 60 rounds, keeping the same shock schedule; if the fastest pre-shock learners still show the largest post-shock degradation after full saturation, the strong-learner/weak-adapter claim is supported, and if they recover as well as the fastest adapter, the reversal was an artifact of the window.","supporting_citations":[],"review_version":1}