{"id":"08de3b6a-515f-4863-9217-ae38ccb5da05","arxiv_id":"2501.14111","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Separate per-node policies reduce the bullwhip effect in a simulated two-node supply chain, but a single shared policy earns more in low-demand settings; SAC beats PPO in high demand.","lead":"This paper tests two ways of letting AI agents manage a factory and a retailer in a supply chain: one shared policy for both, or separate policies that cannot see each other's decisions. The separate policies produce calmer ordering patterns, while a shared policy keeps high inventory and racks up backlogs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Bullwhip-mitigation claim is asserted without the quantitative measure it depends on; the observed order patterns alone do not distinguish mitigation from artifacts of the two-echelon setup.","rationale":"The reader did identify weak empirical support, but focused primarily on convergence and hyperparameter fairness; the bullwhip claim, which is the most emphasized conclusion, is singled out here as the most load-bearing gap because it is explicitly asserted and repeated yet never quantified. This is a correctness risk of the strongest_claim as stated, not merely a matter of statistical rigor. A positive result of the proposed test (heterogeneous retailer bullwhip below 1) would substantially support the central conclusion; a negative result would require curtailing the abstract and conclusion. The reader's other concerns, such as 5-seed evaluation and lack of a baseline policy, are secondary given the code release enables the test. Thus the verdict remains CONDITIONAL with a more targeted condition: the authors should add the bullwhip ratio computation and, if it fails, revise the qualitative claims. No ad hominem or theatrical framing is intended; this is a precise, checkable gap.","tokens_in":14048,"tokens_out":1339,"duration_ms":11491,"concrete_test":"Re-analyse the released code and saved policies: for each architecture, demand scenario, and seed, compute Bullwhip = Var(order)/Var(customer demand) separately at the retailer and factory echelons, using the last 500 steps of each episode as in Figs. 4-5. Report the mean and standard deviation across the 5 seeds, and state whether Bullwhip at the retailer is below 1 for heterogeneous agents and above 1 for homogeneous agents. If instead the retailer-level bullwhip is above 1 in both architectures, the mitigation claim in the abstract and conclusion should be weakened accordingly.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central architectural conclusion, that heterogeneous agents mitigate the bullwhip effect while homogeneous agents do not, is stated qualitatively around Figs. 15-18 and repeated in the abstract and conclusion, but no bullwhip measure (e.g., variance/mean ratio of orders versus demand) is ever computed or reported. The figures show order traces, not amplification ratios. This matters because in this two-echelon chain the factory's demand equals the retailer's order, so any smoothing of retailer orders automatically appears as upstream smoothing; whether that constitutes bullwhip mitigation requires comparing order variance at each echelon against the external demand variance. Moreover, the homogeneous agents' strategy of ordering at maximum capacity is a legitimate (if costly) policy under the reward function, and the claim that heterogeneous agents 'mitigate the bullwhip effect' would require showing lower order variance at the retailer relative to demand, not merely describing the strategy as more 'typical'. The EOQ discussion in Section 5 has the same gap: the statement that the heterogeneous retailer order quantity is 'closer to the EOQ' is not backed by any computed EOQ values or distance metric. Because the headline conclusion inherits directly from this unsupported step, the conditional acceptance should hinge on adding the actual measurement.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies a two-echelon supply chain (one factory, one retailer) under two agent architectures: homogeneous (a single shared policy controlling both nodes) and heterogeneous (separate policies with partial, hidden observations). Both PPO and SAC are trained in high- and low-demand settings, with and without a reward-shaping mechanism that penalizes each agent for the other's stockouts without sharing profits. The main reported findings are that heterogeneous agents adopt lower-inventory ordering strategies and mitigate the bullwhip effect, while homogeneous agents pass it on; that in high demand SAC outperforms PPO and the factory controls the chain; and that in low demand homogeneous agents outperform heterogeneous ones and the retailer controls the chain. The paper also claims, in Section 5, that heterogeneous retailer orders are closer to the EOQ than homogeneous orders.","tokens_in":14262,"tokens_out":3102,"duration_ms":31608,"significance":"The question of whether decentralized, separately learned node policies can outperform a single shared policy in a partially observable supply chain is well motivated and of genuine interest to the multi-agent RL and supply-chain communities. The paper combines two demand regimes, on-policy and off-policy algorithms, both stockout and backlog costs, and a reward-sharing mechanism that avoids profit sharing; it also releases its source code and reports hyperparameter tuning. If the behavioral claims were quantitatively supported, the comparison between homogeneous and heterogeneous architectures across demand regimes would be a useful empirical contribution. However, the headline claims currently rest partly on visual inspection of order traces rather than on the quantitative measures they require.","major_comments":[{"comment":"The central claim that heterogeneous agents mitigate the bullwhip effect while homogeneous agents do not is asserted from visual inspection of order plots, but no bullwhip measure is computed or reported. The paper should report a standard variance ratio, e.g., Var(Q_retailer)/Var(D_customer) and Var(Q_factory)/Var(D_customer), or the cumulative ratio across echelons. Because the factory's demand is the retailer's order, any smoothing of retailer orders automatically produces smoother upstream order traces; demonstrating bullwhip mitigation requires comparing order variance at each echelon against the external demand variance. Without such a computation, the abstract and conclusion overstate what the figures show.","section":"Section 5, Figs. 15–18; Abstract and Conclusion"},{"comment":"The statement that the heterogeneous retailer order quantity is closer to the Economic Order Quantity than the homogeneous order quantity is not supported by any computed EOQ value or distance metric. The surrounding text reports mean inventory levels as percentages of capacity, which do not determine order quantities. The authors should compute Q* from Eq. (16) using the stated demand, ordering cost, holding cost, and stockout cost, and compare it with the empirically observed mean order sizes for the heterogeneous and homogeneous agents. As written, this claim is unfalsifiable from the reported data.","section":"Section 5, Eq. (16)"},{"comment":"The comparative performance claims (SAC over PPO in high demand; homogeneous over heterogeneous in low demand) rest on only 5 seeds per configuration, with no convergence diagnostic and no classical inventory baseline. Since PPO and SAC use different tuned hyperparameter sets and may have different convergence speeds, the observed differences could partly reflect training quality rather than architectural or demand-driven effects. The authors should add a convergence check (e.g., evaluation reward over training with confidence bands, or a statistical test across seeds) and ideally a non-RL baseline policy (e.g., a fixed order-up-to policy), to make the architecture and demand-level comparisons load-bearing.","section":"Sections 3.1 and 4.1; Appendix Tables .5–.6"}],"minor_comments":[{"comment":"The spelling of 'homogenous' is inconsistent with the standard 'homogeneous'; this appears in the abstract, Table 3, and several figure captions.","section":"Throughout"},{"comment":"The objective in Eq. (1) mixes notation: the outer sum is over time t but the inner expression uses 'Sp_i × sum_i Q_i' for total nodal profit, whereas the per-node rewards in Eqs. (13)–(14) multiply price by demand or by the other node's order. Please clarify which expression is the actual per-step objective.","section":"Eq. (1)"},{"comment":"The variable D in Eq. (13) and Q1 in Eq. (14) are not explicitly tied to the state transition in Eq. (11); for clarity, define D_{1,t} and D_{2,t} and state that D_{2,t} = Q_{1,t}.","section":"Eqs. (13)–(14)"},{"comment":"These figures are central to the bullwhip discussion but do not clearly label whether the plotted quantities are orders, inventory levels, or both. Please add explicit axis labels and legends.","section":"Figs. 15–18"},{"comment":"The ordering cost Oc in Eq. (16) is never defined in Table 2 or the text; please define all symbols used in the EOQ formula.","section":"Section 5, Eq. (16)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for this venue and the empirical setup is relevant, but the main qualitative claims need quantitative support before the paper can be accepted. The bullwhip measurement and EOQ comparison are straightforward to add and would not require re-running the full study; the seed/convergence issue may require additional experiments or at least a clearer statistical justification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth knowing: this is a competent, clearly written simulation study comparing homogeneous and heterogeneous MARL policies in a two-echelon supply chain, with code released and a reward-shaping scheme that penalizes a partner's stockout without sharing profit. The modest empirical findings about inventory levels and order behavior are plausible. But the headline claim that heterogeneous agents mitigate the bullwhip effect is not actually measured, and there is a label swap in the inventory table that needs fixing.\n\nThe genuinely new part is the comparison pattern: the same environment with shared versus separate policies, SAC and PPO, high and low demand, with both stockout and backlog penalties, plus a reward-sharing mechanism that does not require profit sharing. That combination is not in the cited prior work, and the Table 1 survey is a useful map. The environment description is careful, hyperparameters are reported, and the released code is a real plus. The profit and inventory comparisons are reasonably consistent: in high demand, SAC beats PPO and the factory earns more; in low demand, homogeneous policies beat heterogeneous and the retailer earns more. Those are useful empirical observations even with only five seeds.\n\nThe soft spot is the bullwhip conclusion. The abstract and conclusion state that heterogeneous agents mitigate the bullwhip effect while homogeneous agents pass it on. The evidence is visual inspection of order and inventory plots. No bullwhip measure, such as order variance relative to demand variance or a variance ratio, is computed. In this two-echelon setup, the factory's demand is simply the retailer's order, so any apparent smoothing of the retailer's order trace automatically appears upstream; whether that counts as mitigation requires comparing against the external demand process. The EOQ discussion has the same gap: the claim that the heterogeneous retailer's order quantity is closer to EOQ is made without showing any computed EOQ value or distance. These are specific, fixable omissions, but they are load-bearing for the paper's main conclusion.\n\nAlso minor: Table 3's columns appear swapped for the homogeneous SAC row. The text says factory inventory is 3.13 and retailer 18.91, but the table lists 18.908 under \"Baseline Factory\" and 3.126 under \"Baseline Retailer.\" That needs correcting. No significance testing is reported, and there is no baseline classical inventory policy, so relative performance is taken somewhat on faith.\n\nBottom line: the paper is a solid candidate after revision. It deserves a serious referee. The referee should require a computed bullwhip measure, a real EOQ comparison, and a fix to the table. Who this is for: supply chain RL practitioners who want architectural guidance. It will not reorient the field, but it is a fair empirical contribution.","headline":"Useful empirical comparison of homogeneous vs heterogeneous MARL in a two-echelon supply chain, but the headline bullwhip claim is asserted without the variance-ratio measurement it depends on.","tokens_in":14773,"tokens_out":3935,"would_cite":false,"duration_ms":35628,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In a two-echelon supply chain, decentralized per-node policies with local observations mitigate the bullwhip effect, while a single shared policy passes it on; the best algorithm and the controlling echelon depend on demand level.","keywords":["multi-agent reinforcement learning","supply chain management","bullwhip effect","heterogeneous agents","inventory control","dynamic pricing","reward shaping","soft actor-critic"],"falsifier":"Train both architectures against a standard base-stock or order-up-to inventory policy as a control, measure the ratio of order variance to demand variance at the factory, and repeat with more seeds; if a shared-policy agent ever shows factory-level variance ratios as low as a heterogeneous agent, the bullwhip-mitigation claim is falsified.","tokens_in":13830,"feed_emoji":"📦","tokens_out":7849,"duration_ms":68104,"temperature":0.7,"pith_summary":"Supply chains are usually run by separate companies that do not share full information, yet most multi-agent reinforcement learning for supply chains assumes a single shared policy controlling every node. This paper asks whether that assumption hides the real behaviour: it builds a factory-retailer supply chain where agents set both order quantities and prices, and compares a single shared policy (homogeneous) with separate per-node policies (heterogeneous) under high and low demand. The paper claims that the two architectures learn genuinely different strategies—the homogeneous retailer hoards inventory and racks up backlog penalties, while the heterogeneous retailer orders in a more conventional sawtooth pattern—and that this makes heterogeneous agents dampen the bullwhip effect while homogeneous agents pass it on. It also claims that the best algorithm and the echelon that controls the chain flip with demand: SAC wins and the factory dominates in high demand, while homogeneous agents win and the retailer dominates in low demand. The implication is that architecture choice and demand regime, not just algorithm, determine who profits and how smoothly the chain runs.","feed_headline":"Separate agents tame the bullwhip effect in supply chains","feed_subtitle":"Per-node policies smooth order swings; a shared policy passes them on. Algorithm and control shift with demand.","key_machinery":"The central object is a two-echelon supply chain (one factory, one retailer) formulated as a hidden Markov decision process, in which each agent chooses an order quantity and a selling price each month. The manipulated variable is observability: homogeneous agents share a single joint observation and a single policy, while heterogeneous agents observe only their own inventory, backlog, stockout level, recent demand, and price, making each node's decision process hidden from the other. The reward function combines sales revenue, inventory holding cost, backlog cost, stockout cost, and purchasing cost, and the paper's reward shaping adds a penalty for the partner's stockout to induce collaboration without profit sharing. PPO and SAC are the two learning algorithms whose policies carry the comparison, and the bullwhip effect, measured as order oscillation between echelons, is the behaviour the architectures are shown to affect differently.","core_discovery":"On the paper's own terms, the discovery is that homogeneous and heterogeneous agent architectures are not interchangeable in a two-echelon supply chain. A homogeneous agent, sharing one observation and action space across the retailer and factory, learns to keep the retailer inventory near capacity, incurring constant backlog penalties in exchange for avoiding costly stockouts, and its ordering strategy amplifies order variance between echelons. A heterogeneous configuration, where each node runs its own policy on only its own stock, backlog, stockout, recent demand, and price, learns a more moderate inventory strategy whose order oscillations shrink from retailer to factory, mitigating the bullwhip effect. The paper further claims that in the high-demand environment SAC agents outperform PPO agents and the factory out-earns the retailer, whereas in the low-demand environment homogeneous agents outperform heterogeneous agents and the retailer out-earns the factory by a wide margin, with the outcome tracking the agents' ability to charge higher prices. Reward shaping that penalizes an agent for its partner's stockout shifts inventory holdings without any profit sharing, but it does not overturn these architecture and demand effects.","pith_inferences":["Editorial inference: Because the paper tunes PPO and SAC separately and uses only five seeds, the demand-dependent reversal (SAC and factory win in high demand; homogeneous agents and retailer win in low demand) may be sensitive to tuning budgets, and a fair-randomization or regret-against-baseline comparison would settle it.","Editorial inference: A natural testable extension is to vary the degree of observability between heterogeneous agents, for example sharing demand forecasts but not inventory levels, and to measure factory order variance; the paper's mechanism predicts monotone amplification as information sharing increases.","Editorial inference: The finding that homogeneous agents charge higher prices in low demand may come from the shared observation giving the single policy coordinated market power; a counterfactual in which heterogeneous agents also see both demand streams would separate observability from architecture.","Editorial inference: Extending the chain to three or more echelons should make the dampening effect of per-node policies more pronounced, since bullwhip amplification grows with chain length; this is the paper's own suggestion and remains untested."],"forward_implications":["In high-demand symbiotic chains, decentralized per-node policies reduce order variance from retailer to factory while a single shared policy amplifies it.","Demand level decides which echelon dominates: the factory in high demand and the retailer in low demand.","SAC is the stronger algorithm in high demand; in low demand the homogeneous shared policy earns more, largely through setting higher prices.","Reward shaping that penalizes a partner's stockout shifts inventory holdings toward the factory without requiring profit sharing, although the effect is modest and demand-dependent.","Heterogeneous agents' retailer order quantities sit closer to the Economic Order Quantity than homogeneous agents', suggesting local policies produce more textbook-conventional ordering behaviour."],"supporting_citations":[{"why":"Defines the bullwhip effect as demand-variance amplification along a supply chain; the paper uses it as the criterion for mitigation.","marker":"Lee et al. (1997)"},{"why":"Solves the beer-game inventory problem with a deep Q-network and profit sharing; the homogeneous, centralized baseline the paper contrasts with heterogeneous agents.","marker":"Oroojlooyjadid et al. (2022)"},{"why":"Analyzes multi-agent reinforcement learning for decentralized inventory control; supplies the decentralized baseline and the profit-sharing mechanism the paper modifies.","marker":"Mousa et al. (2024)"},{"why":"Benchmarks deep RL algorithms in two-echelon inventory control; the homogeneous-agent performance that the paper's homogeneous setup is designed to reproduce.","marker":"Stranieri et al. (2024)"},{"why":"Introduces multi-echelon MARL with adjustable individual and group rewards; the reward-sharing approach that motivates the paper's no-profit-sharing shaping.","marker":"Liu et al. (2022)"},{"why":"Establishes deep RL as workable for multi-echelon inventory management, supporting the paper's RL-based comparison.","marker":"Gijsbrechts et al. (2022)"},{"why":"Models competitive MARL for inventory with shared resources and individual rewards; the competitive scenario the paper extends to a two-echelon chain.","marker":"Ding et al. (2022)"},{"why":"Shows that regular-order policies reduce supply risk and cost; the factory ordering pattern the paper identifies in its agents.","marker":"Hekimoğlu et al. (2018)"}],"fun_headline_variants":["Separate agents tame supply chain bullwhip","Per-node policies smooth order swings","Heterogeneous agents curb bullwhip; shared policies worsen it","Agent architecture flips supply chain performance by demand","One policy per node beats shared policy in supply chains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the PPO and SAC agents were trained to near-optimality with comparably tuned hyperparameters, so the reported differences in profit, inventory, backlog, and bullwhip come from architecture and demand rather than from uneven training or hyperparameter luck.","fun_headline_variants_meta":{"raw":{"variants":["Separate agents tame supply chain bullwhip","Per-node policies smooth order swings","Heterogeneous agents curb bullwhip; shared policies worsen it","Agent architecture flips supply chain performance by demand","One policy per node beats shared policy in supply chains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000674,"raw_usage":{"total_tokens":3116,"prompt_tokens":1038,"completion_tokens":2078,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":2005}},"tokens_in":654,"tokens_out":2078,"duration_ms":13986,"temperature":1.0,"reasoning_tokens":2005,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:21:21.073409+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train both architectures against a standard base-stock or order-up-to inventory policy as a control, measure the ratio of order variance to demand variance at the factory, and repeat with more seeds; if a shared-policy agent ever shows factory-level variance ratios as low as a heterogeneous agent, the bullwhip-mitigation claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the bullwhip effect as demand-variance amplification along a supply chain; the paper uses it as the criterion for mitigation."},{"cited_title":", author Nazari, M","cited_arxiv_id":null,"evidence_quote":"Solves the beer-game inventory problem with a deep Q-network and profit sharing; the homogeneous, centralized baseline the paper contrasts with heterogeneous agents."},{"cited_title":", author van de Berg, D","cited_arxiv_id":null,"evidence_quote":"Analyzes multi-agent reinforcement learning for decentralized inventory control; supplies the decentralized baseline and the profit-sharing mechanism the paper modifies."},{"cited_title":", author Stella, F","cited_arxiv_id":null,"evidence_quote":"Benchmarks deep RL algorithms in two-echelon inventory control; the homogeneous-agent performance that the paper's homogeneous setup is designed to reproduce."},{"cited_title":", author Hu, M","cited_arxiv_id":null,"evidence_quote":"Introduces multi-echelon MARL with adjustable individual and group rewards; the reward-sharing approach that motivates the paper's no-profit-sharing shaping."},{"cited_title":", author Boute, R","cited_arxiv_id":null,"evidence_quote":"Establishes deep RL as workable for multi-echelon inventory management, supporting the paper's RL-based comparison."}],"review_version":1}