{"id":"47b3221d-e2c8-4808-ac08-421567dc4666","arxiv_id":"2506.06368","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using LSTM forecasts as a no-pandemic baseline, the study reports that many U.S. industries showed actual bullwhip amplification during 2020-2023 that the forecasts did not predict.","lead":"This paper measures whether COVID-19 made the bullwhip effect, the amplification of demand swings up supply chains, stronger across 77 U.S. manufacturing, wholesale, and retail industries. It uses machine learning forecasts as a stand-in for what demand would have been without the pandemic, then compares forecasted and actual variance ratios.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The forecast-as-counterfactual baseline is not validated: LSTM point forecasts are selected for MAPE, not for variance preservation, so the false-negative counts in Table 2 may be a forecasting artifact rather than COVID-driven amplification; the paper never compares actual 2016-2019 ratios to…","rationale":"The reader's weakest assumption correctly identifies the forecast-as-counterfactual baseline as the load-bearing issue. The paper's design compares an LSTM forecast (trained on 1992-2015) against actual 2020-2023 data and interprets discrepancies as COVID effects. But a forecasting model is not a causal counterfactual. The model is selected for point forecast accuracy, and its variance properties are never checked. The paper itself notes that 2016-2019 is used only as a benchmark for model performance, not for validating the amplification-ratio baseline. Without an actual-to-actual pre/post comparison or a demonstration that the LSTM reproduces realized amplification ratios in the pre-COVID period, the false-negative counts do not identify pandemic-induced bullwhip. The proposed test directly supplies the missing comparison. Because the paper has useful descriptive value and the authors could address this with additional analyses, the conditional verdict stands unchanged rather than being upgraded or rejected.","tokens_in":14898,"tokens_out":3436,"duration_ms":34980,"concrete_test":"Re-run the bullwhip analysis using actual data only: for each of the 77 industries compute actual amplification ratios for 2016-2019 and 2020-2023 and classify industries into the same four zones using the pre-COVID actual ratio as the baseline. If the false-negative rate in 2020-2023 is not significantly larger than the analogous 'pre-COVID false-negative' rate (e.g., comparing 2013-2016 as baseline vs 2016-2019 actual), the claim that COVID amplified BWE is not supported. Additionally, check LSTM variance preservation in 2016-2019: if Var(forecasted series) is systematically below Var(actual) for demand or production, adjust for this bias before counting false negatives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on treating LSTM forecasts for 2020-2023 as the no-COVID counterfactual (Section 3: 'Forecasted values served as a baseline to represent conditions without the impact of COVID-19'). This is not supported. LSTM is chosen solely by MAPE (Section 4.1), and MAPE rewards accurate point forecasts, not preservation of the variance, autocorrelation, or inventory-demand comovement that determine the amplification ratio in Eq. 12. A mean-reverting or smoothed forecast will mechanically produce different variance ratios from the observed series even in a stable period. The paper never validates that forecasted amplification ratios match actual ratios in 2016-2019, nor does it report actual pre-COVID amplification ratios. Consequently, the 31.2% false-negative rate in Table 2 could reflect the forecast's variance compression or structural model error, not a pandemic-induced bullwhip effect. The conclusion that this is 'empirically demonstrated' overstates what the design can identify.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the bullwhip effect (BWE) across 77 U.S. manufacturing, wholesale, and retail industries using monthly Census data from 1992 to 2023. It forecasts demand and inventory series with SARIMA, Prophet, RNN, and LSTM models, selects LSTM based on MAPE, and then compares the actual amplification ratio for 2020–2023 with the ratio computed from LSTM forecasts. The forecasts are interpreted as a no-COVID counterfactual. Based on the comparison, the paper reports that 31.2% of industries (24 of 77) show a false-negative pattern, i.e., actual bullwhip where forecasts predicted none, and concludes that COVID-19 significantly amplified the BWE, especially among manufacturers.","tokens_in":15123,"tokens_out":4452,"duration_ms":40507,"significance":"If the counterfactual interpretation were valid, the paper would provide a useful industry-level map of BWE during COVID-19 and a forecasting-based screening approach. Strengths include the use of public Census data, the transparent variance-ratio definition in Eq. (12), a multi-model forecasting comparison, and a literature-grounded discussion of demand and supply shocks in Section 4.3. However, the central inference currently rests on an unvalidated assumption that LSTM forecasts represent no-COVID conditions, and the paper provides no statistical test of its headline claim. The significance of the reported results therefore remains conditional until the forecasting baseline is shown to be a valid counterfactual.","major_comments":[{"comment":"The central claim that COVID-19 amplified the bullwhip effect rests on treating the LSTM forecast for 2020–2023 as a no-COVID counterfactual, but this assumption is never validated. Section 3.1 splits the test data into 2016–2019 and 2020–2023, yet the only model selection evidence presented is MAPE in Table 1. MAPE rewards point forecast accuracy and does not guarantee that the forecast preserves the variance of the series, which is the quantity entering Eq. (12). Without a demonstration that forecasted amplification ratios track actual ratios in the stable 2016–2019 period, the 31.2% false-negative rate in Table 2 could be an artifact of variance compression or other forecast error structure rather than a pandemic-induced bullwhip effect. This is a load-bearing gap that must be addressed.","section":"§3, §4.2"},{"comment":"The paper claims in Section 5 that the BWE increase has been 'empirically demonstrated' and the abstract states that COVID-19 'significantly amplified' the BWE, but no statistical test or confidence interval supports this claim. Table 2 reports counts of industries in four zones; even if the counterfactual were valid, a formal comparison (e.g., a paired bootstrap of amplification ratios, a permutation test of the false-negative rate, or a regression with industry fixed effects) would be needed to support the word 'significantly'. As presented, the analysis is purely descriptive and the conclusion overstates what the design can identify.","section":"§4.2, §5"},{"comment":"The pre-COVID period 2016–2019 is used only to compare MAPE, not to validate the bullwhip measurement itself. The paper should report, for the pre-COVID test period, both the actual and the LSTM-forecasted amplification ratios per industry, along with the distribution of their differences. This is the minimal diagnostic needed to separate ordinary forecast error from COVID-specific effects, and it can be computed from data already used in the manuscript.","section":"§3.1, §4.1"}],"minor_comments":[{"comment":"The manufacturer false-positive entry is listed as '1 (0.02%)' but 1 out of 51 manufacturer industries is approximately 1.96%; please correct the percentage and verify all cell counts and totals for consistency.","section":"Table 2"},{"comment":"There is an internal inconsistency in the training window: Section 3.1 states 'training (1992–2015) and test (2016–2023)', while Section 3.2 states 'Monthly data from 1992 to 2016 was used to train forecasting models'. Please clarify the exact split.","section":"§3.1, §3.2"},{"comment":"The LSTM architecture is not specified beyond equations (4)–(9); for reproducibility, please report the number of hidden layers, hidden units, learning rate, batch size, and the tuning procedure referenced in Figure 1.","section":"§3.2.4, §4.1"},{"comment":"The claim that this study is 'the first to employ industry-level data to forecast the BWE' should be qualified, as prior industry-level empirical studies (e.g., Cachon et al. 2007) already use industry-level data; the novelty statement should focus on the forecasting-based counterfactual approach rather than industry-level data per se.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to have been accepted by a journal already, but the scientific content is what we are assessing. The core result is not yet supported because the forecast-as-counterfactual is unvalidated and no inferential statistics are provided. The needed work, however, is within the scope of the existing data and analysis pipeline: reporting pre-COVID forecasted-versus-actual amplification ratios and adding a formal significance test would substantially strengthen the paper. I therefore see this as a major-revision case rather than a rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's what you should know before reading: this paper gives you a broad, clearly organized map of which U.S. industries showed bullwhip during 2020-2023, but its headline claim—that COVID significantly amplified the bullwhip effect—is not actually supported by the analysis as written.\n\nThe good part is real. The authors assemble monthly Census data (M3, MWTS, MARTS) for 77 industries across manufacturing, wholesale, and retail, deflate and margin-adjust it, run four forecasting models, and classify each industry into a four-zone forecast/actual bullwhip table. That is a new empirical exercise at this scale. The industry-by-industry discussion in Section 4.3, connecting false-negative industries to documented demand and supply shocks, is useful for resilience planning. The data are public and the method is described well enough to reproduce.\n\nNow the soft spot, and it is load-bearing. The method treats LSTM forecasts for 2020-2023 as the no-COVID counterfactual because the model was trained on 1992-2015 data. But LSTM was selected for MAPE on 2016-2019. MAPE rewards mean accuracy, not preservation of variance, autocorrelation, or comovement. A smoothed forecast can mechanically lower the variance of forecasted demand and inventory, which directly changes the amplification ratio in Equation 12. The paper never shows that forecasted amplification ratios match actual ratios in the pre-COVID holdout period, and it never reports actual 2016-2019 amplification ratios for comparison. So the 31.2% false-negative rate in Table 2 could be a forecasting artifact rather than a pandemic effect. The conclusion that this is 'empirically demonstrated' overstates what the design can identify.\n\nThat said, this is not a useless paper. The actual amplification ratios during COVID are themselves a useful descriptive fact, and the forecast comparison is a reasonable idea even if it needs validation. A revision that (1) reports actual pre-COVID vs COVID amplification ratios, (2) checks whether forecasted and actual ratios align in 2016-2019, and (3) adds uncertainty intervals would address the core objection. The qualitative shock narratives would then be corroborating context rather than the main evidence.\n\nMy take: this deserves a serious referee because the data work is substantial and the flaw is fixable in revision. I would not cite the causal claim as it stands, but I would keep it on the shelf for the empirical map and as a teaching example of when a counterfactual forecast needs validation.","headline":"Useful industry-level map of where bullwhip appeared during COVID, but the causal claim rests on an unvalidated forecast-as-counterfactual and needs a serious revision.","tokens_in":15619,"tokens_out":3007,"would_cite":false,"duration_ms":26163,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"COVID-19 amplified the bullwhip effect in 31% of U.S. industries","keywords":["Bullwhip effect","COVID-19","Supply chain management","LSTM forecasting","Industry-level analysis","Amplification ratio","U.S. Census data","Panel data"],"falsifier":"Look at LSTM forecast errors in the pre-COVID holdout period 2016–2019: if the model systematically under-predicts variance even before the pandemic, then the 2020–2023 false-negative count is not a clean COVID signal. A concrete check is to compute amplification ratios from forecast residuals for non-pandemic periods and see whether a comparable share of industries would be misclassified as false negatives; if so, the claimed pandemic amplification is partially an artifact of forecast bias.","tokens_in":14705,"feed_emoji":"🏭","tokens_out":5737,"duration_ms":48460,"temperature":0.7,"pith_summary":"The paper tries to establish that the COVID-19 pandemic intensified the Bullwhip Effect—the amplification of demand variability as orders move from retailers to wholesalers to manufacturers—across U.S. industry-level data. Using monthly Census data for 77 industries over 1992–2023, the authors build a counterfactual baseline with LSTM forecasts trained on pre-pandemic data, then compare forecasted variance-amplification ratios to actual ones for 2020–2023. They report that 31.2% of industries show actual bullwhip in the pandemic period where the forecast baseline predicted none, with manufacturers the most affected. The intended payoff is a way to identify which industries are structurally vulnerable to external shocks and to forecast bullwhip risk before a crisis.","feed_headline":"COVID-19 amplified the bullwhip effect in 31% of U.S. industries","feed_subtitle":"An LSTM no-COVID baseline missed 24 industries with real demand swings; manufacturers were hit hardest.","key_machinery":"The load-bearing object is the amplification ratio, $\\mathrm{Var}(\\text{production})/\\mathrm{Var}(\\text{demand})$, computed on log-differenced monthly series, where production for a stage is inferred from the identity $Y_t = S_t + (I_t - I_{t-1})$, shipments plus inventory change. A ratio above 1 indicates bullwhip, and a ratio below 1 indicates variance smoothing. The argument runs on the comparison between actual ratios and ratios computed from LSTM forecasts; the LSTM forecast, trained on 1992–2015 and validated on 2016–2019, serves as the \"no-COVID\" baseline, and industries are classified into four zones depending on whether forecast and actual agree or disagree. The false-negative zone—forecast below 1, actual above 1—is the paper's operational definition of COVID-amplified bullwhip.","core_discovery":"On its own terms, the paper's central discovery is that the COVID-19 shock did not merely amplify demand variability uniformly; it shifted a substantial number of industries across the bullwhip threshold in ways a no-COVID forecast could not anticipate. For 2020–2023 the authors calculate the amplification ratio $\\mathrm{Var}(\\text{production})/\\mathrm{Var}(\\text{demand})$ from actual data and from LSTM-forecast data treated as the counterfactual without COVID. Comparing the two, they find 24 of 77 industries (31.2%) in the \"false negative\" zone—actual bullwhip present but forecast absent—and only 2 false positives. Broken down by stage, 18 of these 24 are manufacturers, and 58.8% of all manufacturing industries exhibit bullwhip in the actual period, consistent with the idea that upstream positions absorb the most amplified demand signal.","pith_inferences":["Editorial inference: the counterfactual reading depends on the LSTM forecast's variance being unbiased in 2020–2023; if the model under-predicts variance for reasons unrelated to the pandemic, the 31.2% false-negative rate could overstate COVID's causal contribution.","Editorial inference: the four-zone classification could be turned into an early-warning score, treating false-negative industries as a training set for features like upstream position, supply shock exposure, and inventory intensity.","Editorial inference: a natural testable extension is to repeat the forecast-baseline comparison on a later non-pandemic window, such as 2024–2026, and check whether the false-negative rate falls back toward pre-COVID levels."],"forward_implications":["COVID-19 raised bullwhip incidence above what pre-pandemic patterns would predict, with 24 U.S. industries crossing the amplification threshold that a no-COVID forecast missed.","Manufacturers bear most of the pandemic-induced bullwhip: 18 of the 24 false-negative industries are manufacturers, and 58.8% of manufacturing industries show actual bullwhip.","Wholesalers are more exposed than retailers, consistent with the bullwhip logic that intermediate and upstream stages face both demand and supply volatility.","Industry-specific demand and supply shocks—plant outbreaks, shutdowns, panic buying, and logistics breakdowns—are the concrete triggers that turn an otherwise predictable industry into a bullwhip case.","Forecasting the amplification ratio during stable periods can flag bullwhip-prone industries before a shock, giving supply chain managers a proactive screening tool."],"supporting_citations":[{"why":"Supplies the industry-level method of measuring bullwhip via variance of inferred production relative to demand, the template for the paper's amplification ratio.","marker":"Cachon et al. (2007)"},{"why":"Underlies the production identity Y = S + (I_t - I_{t-1}) used to infer production from shipments and inventory changes.","marker":"Blinder et al. (1981)"},{"why":"Source for the amplification ratio formula Var(production)/Var(demand) as the bullwhip metric.","marker":"Disney and Towill (2003)"},{"why":"Defines the LSTM architecture whose forecasts constitute the no-COVID counterfactual baseline.","marker":"Hochreiter and Schmidhuber (1997)"},{"why":"Justifies detrending and deseasonalizing before neural network forecasting, a preprocessing step the LSTM baseline relies on.","marker":"Zhang and Qi (2005)"},{"why":"Source of the M3, MWTS, and MARTS monthly industry demand and inventory series for the 77 U.S. industries.","marker":"U.S. Census Bureau, 2024"}],"fun_headline_variants":["COVID-19 triggered bullwhip in 24 U.S. industries, forecast missed","Manufacturers hit hardest as COVID amplified bullwhip effect","No-COVID baseline failed to foresee real bullwhip in 31% of industries","Supply chain shock: COVID-induced bullwhip concentrated in manufacturers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an LSTM trained on 1992–2015 data and validated on 2016–2019 gives a valid picture of what 2020–2023 demand and inventory would have been if COVID-19 had never happened; if the forecast's variance is biased low for unrelated reasons, the false-negative counts overstate the pandemic's effect.","fun_headline_variants_meta":{"raw":{"variants":["COVID-19 triggered bullwhip in 24 U.S. industries, forecast missed","Manufacturers hit hardest as COVID amplified bullwhip effect","No-COVID baseline failed to foresee real bullwhip in 31% of industries","Supply chain shock: COVID-induced bullwhip concentrated in manufacturers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000446,"raw_usage":{"total_tokens":2216,"prompt_tokens":869,"completion_tokens":1347,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":1267}},"tokens_in":485,"tokens_out":1347,"duration_ms":10547,"temperature":1.0,"reasoning_tokens":1267,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:54:38.194543+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Look at LSTM forecast errors in the pre-COVID holdout period 2016–2019: if the model systematically under-predicts variance even before the pandemic, then the 2020–2023 false-negative count is not a clean COVID signal. A concrete check is to compute amplification ratios from forecast residuals for non-pandemic periods and see whether a comparable share of industries would be misclassified as false negatives; if so, the claimed pandemic amplification is partially an artifact of forecast bias.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source for the amplification ratio formula Var(production)/Var(demand) as the bullwhip metric."}],"review_version":1}