REVIEW 3 major objections 5 minor 33 references
Process and Policy Insights from an Intercomparison of Open Electricity System Capacity Expansion Models
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Four open-source capacity expansion models, given identical inputs and configurations, converge on nearly the same system costs—within 0.2–0.3%—so residual portfolio differences are near-random substitutions, not model bias.
desk verdict Genuinely useful intercomparison: the configuration findings are robust and practical, but the 0.2-0.3% cost convergence is partly a product of the harmonization loop and needs honest reframing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The scenario–configuration distinction is the load-bearing separation: scenarios are the policy worlds being optimized (current policies versus net-zero trajectories with carbon buyout prices, transmission limits, and CCS availability), while configurations are the model setup choices (age-based vs economic retirement, myopic vs foresight, 52 vs 20 sampled weather weeks, unit-commitment on/off). The harmonization protocol itself is the central mechanism: a shared data pipeline supplies identical costs, fuel prices, loads, and resource potentials to all four models, and an iterative consensus process aligns structural assumptions to a common base case. Finally, a single operational simulation—a full-year dispatch run with fixed capacities and unit-commitment constraints—provides a common cost yardstick, so the cost of any model's portfolio is evaluated in the same way. Together these pieces let the paper attribute differences to configuration rather than data or solver noise.
What would settle it
Take the four models out of the harmonization loop: run each with its own native default inputs and configurations on the same net-zero scenario, and compare system costs using a single common operational simulation. If costs diverge by much more than the reported 0.2–0.3% or capacity choices line up with model identity, the convergence is an artifact of the iterative revision process rather than a property of the models. A complementary check is to apply the same harmonized protocol to a new scenario that was not part of the alignment process (for example, high electrification demand or strict zero-emission compliance with no buyout) and see whether agreement stays below 1%.
Extended reading notes
Core claim
The central claim is that when four structurally different open-source capacity expansion models are forced onto a common footing—same data, same policy scenarios, same configuration choices, same solver settings—they converge on nearly the same objective value: net present system costs differ by only 0.2–0.3% across models for the current-policy and net-zero scenarios, and by under 1% for most configurations. The authors interpret this as evidence that all four models are finding essentially equivalent global cost minima, so the visible differences in capacity and generation choices are normal substitutions among plans with similar costs, not persistent biases of any one model. They then show that configuration features—most strongly the choice between age-based and economic retirement of existing plants, and secondarily unit-commitment constraints and the number of sampled weather weeks—move system costs and investment patterns more than the identity of the model does. The paper presents this as a demonstration that model intercomparison with strictly harmonized inputs can isolate structural differences, provide reliable policy insight, and identify which model features matter most for a given scenario.
Load-bearing premise
The models were revised repeatedly during the harmonization process until residual differences looked small, so the headline cost convergence is partly a product of the harmonization protocol rather than an independent measurement of how close the models would naturally be.
Editorial extensions
If this is right
- If harmonized open models converge, consensus findings from multi-model studies can guide policy even when the models differ in internal implementation.
- Configuration choices—especially economic retirement and unit-commitment constraints—are more influential than model choice, so model users should document and align these choices carefully.
- Under current policies, the models consistently find that emissions do not fall fast enough; only a carbon buyout price of $1,000/tonne yields steady reductions in the net-zero scenario.
- Restricting all inter-regional transmission expansion raises 2050 costs by roughly 4.6% and emissions by 55%, showing that transmission is valuable but can be substituted with local renewable build-out plus batteries.
- The 52-week myopic base configuration often beats the 20-week foresight configuration on cost, suggesting that temporal detail can matter more than inter-temporal foresight in these models.
Reading between the lines
- The paper's own appendix reports an erroneous 2027 emissions target (847 million tonnes) inherited from the reference trajectory, likely a typo for 494 million; because all four models shared the same target, it does not by itself upset the convergence result, but it shows how a single input error can propagate uniformly through harmonized models.
- If the convergence holds beyond the tested scenarios, practical planning insight will come more from improving data quality and feature fidelity (retirement rules, temporal sampling, unit commitment) than from building new models, since model identity is not the main source of variation.
- A testable extension would feed the same harmonized inputs to a structurally different model family, such as an equilibrium-based or production-cost model, to see whether the cost-minimum agreement persists; that would separate the influence of shared data from the influence of shared optimization logic.
- The study's result that a 20-week foresight model does not beat a 52-week myopic model suggests a caution for common practice; a systematic sweep of sample-week counts within a single model could identify where temporal detail outweighs intertemporal foresight.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a structured intercomparison of four open-source capacity expansion models (GenX, Switch, TEMOA, USENSYS) for the continental US power system, using PowerGenome to harmonize inputs and explicit scenarios/configurations. The authors claim that after iterative harmonization, the models produce very similar capacity portfolios and system costs (0.2–0.3% NPV spread in the base case, <1% for most configurations), interpret residual differences as quasi-random variation, and then use the harmonized models to study policy-relevant scenario and configuration effects (carbon buyout prices, transmission constraints, CCS availability, retirement rules, temporal sampling, foresight). The paper also reflects on lessons for future intercomparison efforts.
Significance. If the convergence claim is valid, the paper provides a strong practical result: harmonized open-source capacity expansion models can agree on system costs to within a fraction of a percent, making configuration effects the dominant source of divergence. The study is valuable as a community resource: it is transparent about the harmonization protocol, uses a common operational simulation as a cost metric, makes results and data available via GitHub, and candidly documents an input data error in Appendix C. The configuration-effect findings (e.g., economic retirement and unit commitment are influential) are useful and less affected by the circularity concern. The main risk is that the headline convergence result is partly an artifact of the iterative harmonization stopping rule, which weakens the paper's central claim as currently framed.
major comments (3)
- [Section 2 (Harmonizing models for the net-zero scenario and base configuration) and Section 3.1] The 0.2–0.3% cost spread in Section 3.1 is measured after the iterative harmonization loop described in Section 2, whose stopping rule is that residual differences in energy mix, infrastructure builds, and total costs 'did not show large, unexplainable differences across models.' The convergence claim is therefore partly an output of the protocol: models were revised until agreement was judged adequate, and then agreement was reported as evidence that the models find 'nearly equivalent global cost minimums.' To make the claim load-bearing, the paper should either report results from the un-revised models, quantify how much each revision changed system costs and portfolios, or otherwise demonstrate that the stopping rule did not suppress structural diversity.
- [Section 2 ('Calculating model costs') and Section 3.1] The NPV spread is computed by evaluating each model's proposed portfolio in a single operational model (GenX), not by comparing each model's native objective values. This is a sensible common metric, but it supports a narrower claim: the portfolios are nearly cost-equivalent under GenX's dispatch, cycling, and penalty assumptions. It does not establish that the models' native optimization problems share a global minimum, and residual capacity differences could reflect the evaluator compressing cost differences among divergent portfolios. I recommend softening 'global cost minimums' to 'cost-equivalent under a common operational metric' and, if feasible, adding a robustness check with a second evaluator or reporting native objective values.
- [Appendix C] The 2027 CO2 target used in all net-zero runs is admitted to be the 2025 REPEAT value (847 Mt) rather than the true 2027 value (494 Mt). Because the net-zero scenario's near-term trajectory is central to Section 3.2.1 and Figure 2, and because the trajectory's steepness affects investment timing in Figures 1, 4, and 5, this error is not merely cosmetic. The authors should correct the target and rerun the net-zero scenario (or clearly demonstrate that all qualitative conclusions are unchanged under the corrected 2027 cap), and should mark the corrected table in Appendix C.
minor comments (5)
- [Abstract and Section 3.1] The abstract states 'less than 1% difference in system costs for most configurations,' but Section 3.1 reports 0.2–0.3% for the base configuration only; please specify the exact set of configurations and which cost metric is used for the abstract's claim.
- [Figure 6 and Section 3.3] The legend uses 'Age' and 'Economic' in a way that may be confused with the retirement-type markers; consider a clearer legend that separates retirement type, temporal sampling, and period type.
- [Appendix C, Table C1] Table C1 labels the third column 'REPEAT' but the 2027 row says '847 (in 2025)'; once corrected, add a footnote explaining the correction so readers do not use the old trajectory.
- [Section 3.2.2 and Discussion item (iv)] The 4.6% cost increase and 55% emissions increase for no transmission expansion are stated consistently, but Figure 3's panel labels could clarify whether 'costs' are annual operational costs or total system costs.
- [Section 2 (Harmonizing models for additional scenarios and configurations) and Table 5] The statement that 'most or all of the models were extended and adapted as needed' is relevant to interpretation of configuration effects; please state explicitly in Table 5 which models could not implement which configurations.
Circularity Check
Convergence claim is the harmonization stopping rule, not an independent measurement
-
fitted input called prediction
[Section 2 'Harmonizing models for the net-zero scenario and base configuration'; Section 3.1 'Demonstration of Harmonization']
"System costs from the operational simulation were used to confirm that the models solved the same problem to within the desired level of tolerance. Harmonization was considered sufficient when any residual differences in results... did not show large, unexplainable differences across models. ... The net present values of the operational model system costs for the current policy and net-zero scenarios vary by 0.2-0.3% across the models, demonstrating that models are all finding nearly equivalent global cost minimums."
The 0.2-0.3% NPV spread is the same quantity used as the harmonization stopping rule: models were iteratively revised until residual cost and energy-mix differences were judged 'not large,' and then that residual spread is reported as evidence that the models find nearly equivalent minima. The convergence claim is therefore an output of the harmonization protocol's tolerance, not an independent test. Additionally, the cost metric is a single GenX operational simulation applied to every model's portfolio, so the spread reflects GenX's evaluation of foreign portfolios rather than the proximity of each model's native objective values. The subsequent inference that capacity differences are 'quasi-random variation' inherits the same circularity.
full rationale
This analysis identifies one load-bearing circular step: the headline convergence result is the stopping rule of the harmonization loop, renamed as a finding. The 0.2-0.3% NPV spread in Section 3.1 is measured after iterative model revisions whose sufficiency criterion was exactly that residual cost and energy-mix differences be 'not large,' and the same GenX operational-simulation cost metric was used both to confirm harmonization and to demonstrate convergence. The spread is therefore an output of the protocol's tolerance, not an independent measurement of model proximity. I do not count the GenX evaluator as a separate circular step; it is a validity concern that compounds the circularity of the metric. The configuration-effect findings (economic retirement, unit commitment, foresight vs myopic, etc.) are comparative within and across models and are less affected, so the paper is only partially circular. Appendix C admits an error in the 2027 CO2 target (847 vs REPEAT 494 Mt); that is a correctness issue, not circularity, and does not change the score. Self-citations to PowerGenome and REPEAT are inputs, not load-bearing circular justifications.
Assumptions & free parameters
free parameters (6)
- Weighted average cost of capital (WACC) =
5%
- Foresight discount rate =
2%
- Unserved load penalty =
$5,000/MWh
- Carbon buyout price =
$200/tonne base; $50 and $1,000 in child scenarios
- Hydrogen fuel price =
$16/MMBTU
- 2027 CO2 emissions target =
847 million tonnes, erroneously based on 2025 REPEAT value
assumptions (5)
- domain assumption Least-cost optimization by each model approximates efficient power system planning.
- domain assumption Representative weeks capture the weather and load variability needed for capacity decisions.
- domain assumption The GenX operational simulation provides an unbiased common cost metric for all models.
- ad hoc to paper The iterative harmonization stopping rule is sufficient for claiming convergence.
- domain assumption PowerGenome input data are sufficiently accurate for all models.
Cite this review
Pith. "Pith review of Process and Policy Insights from an Intercomparison of Open Electricity System Capacity Expansion Models." pith.science (2026). https://pith.science/paper/PGDM3E5H
@misc{pith2026241113783,
author = {Pith},
title = {Pith review of: Process and Policy Insights from an Intercomparison of Open Electricity System Capacity Expansion Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/PGDM3E5H}},
note = {Machine review of arXiv:2411.13783}
}
read the original abstract
This study performs a detailed intercomparison of four open-source electricity capacity expansion models - Temoa, Switch, GenX, and USENSYS - to evaluate 1) how closely the results of these models align when inputs and configurations are harmonized, and 2) the degree to which varying model configurations affect outputs. We harmonize the inputs to each model using PowerGenome and use clearly defined scenarios (policy conditions) and configurations (model setup choices). This allows us to isolate how differences in model structure affect policy outcomes and investment decisions. Our framework allows each model to be tested on identical assumptions for policy, technology costs, and operational constraints, allowing us to focus on differences that arise from inherent model structures. Key findings highlight that, when harmonized, models produce very similar capacity portfolios under current policies and net-zero scenarios, with less than 1% difference in system costs for most configurations. This agreement among models allows us to focus on how configuration choices affect model results. For instance, configurations with unit commitment constraints or economic retirement yield different investments and system costs compared to simpler configurations. Our findings underscore the importance of aligning input data and transparently defining scenarios and configurations to provide robust policy insights.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Intergovernmental Panel on Climate Change (IPCC) 2022 Summary for Policymakers Global Warming of 1.5 °C: IPCC Special Report on Impacts of Global Warming of 1.5 °C above Pre-industrial Levels in Context of Strengthening Response to Climate Change, Sustainable Development, and Efforts to Eradicate Poverty ed Intergovernmental Panel on Climate Change (IPCC)...
work page 2022
-
[2]
Deep Decarbonization Pathways Project 2015 Pathways to deep decarbonization 2015 synthesis report Tech. rep. SDSN - IDDRI
work page 2015
-
[3]
catf.us/2020/10/state-and-regional-decarbonization-commitments/
Clean Air Task Force 2020 State and Utility Decarbonization Commitments URL https://www. catf.us/2020/10/state-and-regional-decarbonization-commitments/
work page 2020
-
[4]
Huntington H G, Weyant J P and Sweeney J L 1982 Omega 10 449–462 publisher: Elsevier
work page 1982
-
[5]
Henry C L, Eshraghi H, Lugovoy O, Waite M B, DeCarolis J F, Farnham D J, Ruggles T H, Peer R A M, Wu Y, de Queiroz A, Potashnikov V, Modi V and Caldeira K 2021 Applied Energy 304 117745 ISSN 0306-2619 URL https://www.sciencedirect.com/science/article/pii/ S0306261921010916
work page 2021
-
[6]
van Ouwerkerk J, Gils H C, Gardian H, Kittel M, Schill W P, Zerrahn A, Murmann A, Launer J, Torralba-D ´ ıaz L and Bußar C 2022 Renewable and Sustainable Energy Reviews 157 112004 ISSN 1364-0321 URL https://www.sciencedirect.com/science/article/pii/ S1364032121012673
work page 2022
-
[7]
S´ anchez-P´ erez P A, Staadecker M, Szinai J, Kurtz S and Hidalgo-Gonzalez P 2022Applied Energy 317 119022 ISSN 0306-2619 URL https://www.sciencedirect.com/science/article/pii/ S0306261922004275
-
[8]
Jacobson A F, Mauzerall D L and Jenkins J D 2024 Environmental Research: Energy 1 035009 ISSN 2753-3751 publisher: IOP Publishing URL https://dx.doi.org/10.1088/2753-3751/ ad6d6f
Show all 33 references
-
[9]
Hoffmann M, Schyska B U, Bartels J, Pelser T, Behrens J, Wetzel M, Gils H C, Tang C F, Tillmanns M, Stock J, Xhonneux A, Kotzur L, Praktiknjo A, Vogt T, Jochem P, Linßen J, Weinand J M and Stolten D 2024 Advances in Applied Energy 16 100190 ISSN 2666-7924 URL https://www.scien...
2024
-
[10]
Hunter K, Sreepathi S and DeCarolis J F 2013 Energy Economics 40 339–349 ISSN 0140-9883 URL https://www.sciencedirect.com/science/article/pii/S014098831300159X
2013
-
[11]
Johnston J, Henriquez-Auba R, Maluenda B and Fripp M 2019 SoftwareX 10 ISSN 2352- 7110 publisher: Elsevier URL https://www.softxjournal.com/article/S2352-7110(18) 30154-7/fulltext
2019
-
[12]
Jenkins J, Sepulveda N, Mallapragada D, Patankar N, Schwartz A, Schwartz J, Chakrabarti S, Xu Q, Morris J and Sepulveda N 2024 GenX URL https://zenodo.org/records/10910030
2024
-
[13]
Lugovoy O 2024 United States Energy System Optimization Model (USENSYS) URL https: //www.usensys.org
2024
-
[14]
org/records/11194213
Schivley G, Welty E, Patankar N, Jacobson A, Xu Q, Manocha A, Pecora B, Bhandarkar R, Jenkins J D and Fripp M 2024 PowerGenome/PowerGenome: v0.6.3 URL https://zenodo. org/records/11194213
2024
-
[15]
eia.gov/outlooks/aeo/
US Energy Information Administration 2023 Annual Energy Outlook 2023 URL https://www. eia.gov/outlooks/aeo/
2023
-
[16]
Selvans Z A, Gosnell C M, Sharpe A, Winter S, Norman B, Bush T, Schira Z and Lamb K 2022 PUDL Data Release v2022.11.30 URL https://zenodo.org/records/7472137
2022
-
[17]
Energy Information Administration (EIA) URL https://www.eia.gov/electricity/data/eia860m/index.php
US Energy Information Administration Preliminary Monthly Electric Generator Inventory (based on Form EIA-860M as a supplement to Form EIA-860) - U.S. Energy Information Administration (EIA) URL https://www.eia.gov/electricity/data/eia860m/index.php
-
[18]
Vibrant Clean Energy, LLC 2024 US Wind and Solar Profiles - 2012 Weather Year URL https://zenodo.org/records/14194558 Intercomparison of Open Electricity System Capacity Expansion Models 29
2024
-
[19]
Mai T T, Jadun P, Logan J S, McMillan C A, Muratori M, Steinberg D C, Vimmerstedt L J, Haley B, Jones R and Nelson B 2018 Electrification Futures Study: Scenarios of Electric Technology Adoption and Power Consumption for the United States Tech. Rep. NREL/TP- 6A20-71500 Nationa...
2018
-
[20]
Jenkins J, Mayfield E, Farbes J, Schivley G, Patankar N and Jones R 2023 Climate Progress and the 117th Congress: The Impacts of the Inflation Reduction Act and Infrastructure Investment and Jobs Act URL DOI:10.5281/zenodo.8087805
2023 doi
-
[21]
nrel.gov/electricity/2023/index
National Renewable Energy Laboratory Annual Technology Baseline 2023 URL https://atb. nrel.gov/electricity/2023/index
2023
-
[22]
US Energy Information Administration 2020 Cost and Performance Characteristics of New Generating Technologies, Annual Energy Outlook 2020 URLhttps://www.eia.gov/outlooks/ archive/aeo20/assumptions/pdf/table_8.2.pdf
2020
-
[23]
Reiner lemoine institute, berlin
Christidis A, Wasike-Schalling A and Arriens J 2023 H2-ready gas-fired power plants. Reiner lemoine institute, berlin
2023
-
[24]
Patankar N, Sarkela-Basset X, Schivley G, Leslie E and Jenkins J 2023 Energy and Climate Change 4 100107 ISSN 2666-2787 URL https://www.sciencedirect.com/science/article/ pii/S2666278723000144
2023
-
[25]
National Energy Technology Laboratory 2022 FECM/NETL CO2 Transport Cost Model (2022) URL https://netl.doe.gov/energy-analysis/details
2022
-
[26]
Internal Revenue Service 2024 Inflation Reduction Act of 2022 URL https://www.irs.gov/ inflation-reduction-act-of-2022
2024
-
[27]
Bistline J, Blanford G, Brown M, Burtraw D, Domeshek M, Farbes J, Fawcett A, Hamilton A, Jenkins J, Jones R, King B, Kolus H, Larsen J, Levin A, Mahajan M, Marcy C, Mayfield E, McFarland J, McJeon H, Orvis R, Patankar N, Rennert K, Roney C, Roy N, Schivley G, Steinberg D, Vict...
2023 doi
-
[28]
Bistline J E T, Brown M, Domeshek M, Marcy C, Roy N, Blanford G, Burtraw D, Farbes J, Fawcett A, Hamilton A, Jenkins J, Jones R, King B, Kolus H, Larsen J, Levin A, Mahajan M, Mayfield E, McFarland J, McJeon H, Orvis R, Patankar N, Rennert K, Robson S, Roney C, Russell E, Schi...
2023 doi
-
[29]
US Environmental Protection Agency 2023 Report on the Social Cost of Greenhouse Gases: Estimates Incorporating Recent Scientific Advances URL https://www.epa.gov/ environmental-economics/scghg
2023
-
[30]
Howard P and Sylvan D 2024 Consensus on Carbon Dioxide Removal: A Large-Sample Expert Elicitation on the Future of CDR Tech. rep. Institute for Policy Integrity URL https: //policyintegrity.org/publications/detail/consensus-on-carbon-dioxide-removal
2024
-
[31]
Zheng R, Schivley G, Hidalgo-Gonzalez P, Fripp M and Roberts M J 2024Working Papers Number: 2024-2 Publisher: University of Hawaii Economic Research Organization, University of Hawaii at Manoa URL https://ideas.repec.org//p/hae/wpaper/2024-2.html
2024
-
[32]
Environmental Protection Agency 2003 Integrated Planning Model URL https://www.epa.gov/ power-sector-modeling/2023-reference-case
2003
-
[33]
Schivley G 2024 MIP results comparison URL https://github.com/gschivley/MIP_results_ comparison
2024
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.