Pith. sign in

REVIEW 2 minor 46 references

modelimportance: An R package for evaluating model importance within a multi-model ensemble

T0 review · 0 major / 2 minor · reviewed 2026-06-28 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The modelimportance R package quantifies each model's contribution to ensemble forecast accuracy.

desk verdict The paper describes an R package for computing model contributions in ensembles but supplies no validation or examples of the metrics. read the letter →

arxiv 2605.30278 v2 pith:LRZOSED7 submitted 2026-05-28 stat.CO

Minsu Kim , Li Shandross , Evan L. Ray , Nicholas G. Reich This is my paper · ORCID
classification stat.CO
keywords ensembleforecastingmodelimportanceRpackageforecastaccuracypointforecastsprobabilisticmulti-modelensembles
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents an R package that measures how much each individual model adds to the accuracy of a combined forecast made from multiple models. Ensembles are widely used because they tend to be more accurate and stable than any single model, yet without ways to score each component's value it is difficult to decide which models to retain or how to refine the group. The package computes importance scores for both point forecasts and probabilistic forecasts, supports several methods for combining models, and includes several different importance metrics. It also provides options to manage cases where some models have no prediction for a given time point. These functions let users build stronger ensembles while learning which models drive the gains.

What carries the argument

The modelimportance package, which applies chosen importance metrics to score each model's effect on ensemble accuracy.

What would settle it

A test in which models the package ranks as low importance are removed from the ensemble and the resulting accuracy does not improve or declines.

Watch

Extended reading notes

Core claim

The modelimportance package quantifies how each component model contributes to the accuracy of ensemble performance for both point and probabilistic forecasts. The package supports multiple ensemble methods and multiple model importance metrics, and it supplies customizable options for handling missing values.

Load-bearing premise

The chosen importance metrics and ensemble methods produce reliable and interpretable measures of each model's value.

Editorial extensions

If this is right

  • Users can identify which models contribute most to ensemble accuracy.
  • The tools support construction of more effective ensembles across forecasting tasks.
  • Analysis yields insights into the role played by each model inside the ensemble.
  • Options for missing values allow the methods to work with incomplete real data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Importance scores could be used to assign different weights to models when forming new ensembles.
  • The same metrics might be applied to combined systems outside forecasting, such as stacked predictors in other domains.
  • Direct comparisons of the package's different metrics on the same data could show which one best matches actual accuracy gains.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 2 minor

Summary. The manuscript describes the R package modelimportance, which supplies implemented tools to quantify the contribution of each component model to ensemble accuracy for both point and probabilistic forecasts. The package supports multiple ensemble methods and model-importance metrics, includes options for handling missing values, and integrates with the hubverse framework for collaborative modeling.

Significance. If the described functionality is correctly implemented, the package would serve as a practical, reusable tool for researchers analyzing ensembles in forecasting applications. Its hubverse compatibility could streamline workflows in collaborative hubs and support post-hoc interpretation of model value without requiring users to code custom importance calculations.

minor comments (2)
  1. The manuscript would benefit from a short table or enumerated list in the main text that explicitly names the supported ensemble methods and importance metrics (currently only alluded to in the abstract).
  2. A minimal worked example (code snippet plus output) illustrating a typical call and interpretation of results would clarify usage for readers who have not yet installed the package.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their supportive review of the modelimportance package manuscript and for recommending minor revision. No major comments were provided in the report.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity in software description

full rationale

This is a software package description paper whose content is limited to documenting implemented functionality, options for missing values, and integration with the hubverse framework. No mathematical derivations, first-principles results, fitted parameters presented as predictions, or uniqueness theorems appear in the text. The central claim is simply that the package supplies the described tools; this claim is not derived from any prior result within the paper and does not reduce to a self-citation chain or definitional loop. The derivation chain is therefore empty and self-contained.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

This is a software-package announcement with no mathematical claims, derivations, or new theoretical constructs; therefore the ledger contains no entries.

how reviews work

0 comments
Cite this review

Pith. "Pith review of modelimportance: An R package for evaluating model importance within a multi-model ensemble." pith.science (2026). https://pith.science/paper/LRZOSED7

@misc{pith2026260530278,
  author       = {Pith},
  title        = {Pith review of: modelimportance: An R package for evaluating model importance within a multi-model ensemble},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LRZOSED7}},
  note         = {Machine review of arXiv:2605.30278}
}
read the original abstract

Ensemble forecasts are commonly used to support decision-making and policy planning across various fields because they often offer improved accuracy and stability compared to individual models. As each model has its own unique characteristics, understanding and measuring the value of each constituent model can support the construction of effective ensembles. The R package modelimportance provides tools to quantify how each component model contributes to the accuracy of ensemble performance for both point and probabilistic forecasts. The package supports multiple ensemble methods and multiple model importance metrics. Additionally, the software offers customizable options for handling missing values. These features enable the package to serve as a versatile tool for researchers and practitioners. It helps not only in constructing an effective ensemble model across a wide range of forecasting tasks, but also in understanding the role of each model within the ensemble and gaining insights into individual models themselves. This package follows the 'hubverse' framework, which is a collection of open-source software, tools and data standards developed to promote collaborative modeling hub efforts and simplify their setup and operation. Doing so enables seamless integration and flexibility with other forecasting tools and systems, allowing many analyses to be performed on existing hubs.

Figures

Figures reproduced from arXiv: 2605.30278 by the authors.

Figure 1
Figure 1. Example plot of three distributional forecasts corresponding to the model output for incident [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Conceptual illustration of measuring model importance in a three-model setting. Each circle [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Comparison of weights assigned to a subset. The plot shows the weights assigned to a subset [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Plot of three point forecasts (median) and the eventually observed values from the [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Runtime to compute model importance scores for point predictions by number of ensemble [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: Runtime to compute model importance scores for point predictions by number of ensemble [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 1 canonical work pages

  1. [1]

    A Unifying Framework for Parallel and Distributed Processing in R using Futures.The R Journal, 13(2):208–227, 2021

    Henrik Bengtsson. A Unifying Framework for Parallel and Distributed Processing in R using Futures.The R Journal, 13(2):208–227, 2021

    2021

  2. [2]

    Evaluating forecasts with scoringutils in R.arXiv preprint arXiv:2205.07090v2, 2024

    Nikos I Bosse, Hugo Gruson, Anne Cori, Edwin van Leeuwen, Sebastian Funk, and Sam Abbott. Evaluating forecasts with scoringutils in R.arXiv preprint arXiv:2205.07090v2, 2024

    work page arXiv 2024

  3. [3]

    Ray, Tilmann Gneiting, and Nicholas G

    Johannes Bracher, Evan L. Ray, Tilmann Gneiting, and Nicholas G. Reich. Evaluating epidemic forecasts in an interval format.PLOS Computational Biology, 17(2):e1008618, 2021

    2021

  4. [4]

    Nonmechanistic forecasts of seasonal influenza with iterative one-week-ahead distributions.PLOS Computational Biology, 14(6):e1006134, 2018

    Logan C Brooks, David C Farrow, Sangwon Hyun, Ryan J Tibshirani, and Roni Rosenfeld. Nonmechanistic forecasts of seasonal influenza with iterative one-week-ahead distributions.PLOS Computational Biology, 14(6):e1006134, 2018

    2018

  5. [5]

    R package version 1.7.7.1

    Tianqi Chen, Tong He, Michael Benesty, Vadim Khotilovich, Yuan Tang, Hyunsu Cho, Kailong Chen, Rory Mitchell, Ignacio Cano, Tianyi Zhou, Mu Li, Junyuan Xie, Min Lin, Yifeng Geng, Yutian Li, and Jiaming Yuan.xgboost: Extreme Gradient Boosting, 2024. R package version 1.7.7.1

    2024

  6. [6]

    The hubverse: open tools for collaborative forecasting, 2024

    Consortium of Infectious Disease Modeling Hubs. The hubverse: open tools for collaborative forecasting, 2024. Accessed: 2025-01-14

    2024

  7. [7]

    Understanding global feature contributions with 25 additive importance measures.Advances in neural information processing systems, 33:17212– 17223, 2020

    Ian Covert, Scott M Lundberg, and Su-In Lee. Understanding global feature contributions with 25 additive importance measures.Advances in neural information processing systems, 33:17212– 17223, 2020

    2020

  8. [8]

    The United States COVID-19 forecast hub dataset.Scientific data, 9(1):462, 2022

    Estee Y Cramer, Yuxin Huang, Yijin Wang, Evan L Ray, Matthew Cornell, Johannes Bracher, Andrea Brennen, Alvaro J Castro Rivadeneira, Aaron Gerding, Katie House, et al. The United States COVID-19 forecast hub dataset.Scientific data, 9(1):462, 2022

    2022

Show all 46 references
  1. [9]

    Flu scenario modeling hub, 2024

    Flu Scenario Modeling Hub. Flu scenario modeling hub, 2024. Accessed: 2025-07-25

    2024

  2. [10]

    Weather forecasting with ensemble methods.Science, 310(5746):248–249, 2005

    Tilmann Gneiting and Adrian E Raftery. Weather forecasting with ensemble methods.Science, 310(5746):248–249, 2005

    2005

  3. [11]

    Strictly Proper Scoring Rules, Prediction, and Estima- tion.Journal of the American Statistical Association, 102(477):359–378, 2007

    Tilmann Gneiting and Adrian E Raftery. Strictly Proper Scoring Rules, Prediction, and Estima- tion.Journal of the American Statistical Association, 102(477):359–378, 2007

    2007

  4. [12]

    R package version 0.1.4

    Ben Hamner and Michael Frasco.Metrics: Evaluation Metrics for Machine Learning, 2018. R package version 0.1.4

    2018

  5. [13]

    Springer Series in Statistics

    Trevor Hastie, Robert Tibshirani, and Jerome Friedman.The Elements of Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001

    2001

  6. [14]

    Financial Time Series Fore- casting with the Deep Learning Ensemble Model.Mathematics, 11(4), 2023

    Kaijian He, Qian Yang, Lei Ji, Jingcheng Pan, and Yingchao Zou. Financial Time Series Fore- casting with the Deep Learning Ensemble Model.Mathematics, 11(4), 2023

    2023

  7. [15]

    R package version 3.2.0

    Jim Hester, Florent Angly, Russ Hyde, Michael Chirico, Kun Ren, Alexander Rosenstock, and Indrajeet Patil.lintr: A ’Linter’ for R Code, 2025. R package version 3.2.0

    2025

  8. [16]

    Evaluating probabilistic forecasts with scoringRules.Journal of Statistical Software, 90(12):1–37, 2019

    Alexander Jordan, Fabian Kr¨ uger, and Sebastian Lerch. Evaluating probabilistic forecasts with scoringRules.Journal of Statistical Software, 90(12):1–37, 2019

    2019

  9. [17]

    Guerra, Sophie A

    Jordan A. Guerra, Sophie A. Murray, D. Shaun Bloomfield, and Peter T. Gallagher. Ensemble forecasting of major solar flares: methods for combining models.J. Space Weather Space Clim., 10:38, 2020

    2020

  10. [18]

    Co- ordinating collaborative infectious disease modeling projects with the hubverse.medRxiv, 2025

    Melissa Kerr, Rebecca Borchering, Alvaro Castro Rivadeneira, Lucie Contamin, Sebastian Funk, Harry Hochheiser, Emily Howerton, Anna Krystalli, Li Shandross, Nicholas G Reich, et al. Co- ordinating collaborative infectious disease modeling projects with the hubverse.medRxiv, 2025

    2025

  11. [19]

    Ray, and Nicholas G

    Minsu Kim, Evan L. Ray, and Nicholas G. Reich. Beyond forecast leaderboards: Measuring individual model importance based on contribution to ensemble accuracy.International Journal of Forecasting, 2026

    2026

  12. [20]

    R package version 0.6.0.9000, commit 71964f7cd6f79e48f3038952fed8f742a7fe6a5c

    Anna Krystalli and Li Shandross.hubUtils: Core ’hubverse’ Utilities, 2025. R package version 0.6.0.9000, commit 71964f7cd6f79e48f3038952fed8f742a7fe6a5c

    2025

  13. [21]

    Building Predictive Models in R Using the caret Package.Journal of Statistical Software, 28(5):1–26, 2008

    Max Kuhn. Building Predictive Models in R Using the caret Package.Journal of Statistical Software, 28(5):1–26, 2008

    2008

  14. [22]

    Classification and Regression by randomForest.R News, 2(3):18–22, 2002

    Andy Liaw and Matthew Wiener. Classification and Regression by randomForest.R News, 2(3):18–22, 2002

    2002

  15. [23]

    Arkadiusz Lipiecki, Bartosz Uniejewski, and Rafa/suppress l Weron. Postprocessing of point predictions for probabilistic forecasting of day-ahead electricity prices: The benefits of using isotonic distribu- tional regression.Energy Economics, 139:107934, 2024. 26

    2024

  16. [24]

    PostForecasts

    Arkadiusz Lipiecki and Rafa/suppress l Weron. PostForecasts. jl: A Julia package for probabilistic forecast- ing by postprocessing point predictions.SoftwareX, 31:102200, 2025

    2025

  17. [25]

    From local explanations to global understanding with explainable AI for trees.Nature machine intelligence, 2(1):56–67, 2020

    Scott M Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. From local explanations to global understanding with explainable AI for trees.Nature machine intelligence, 2(1):56–67, 2020

    2020

  18. [26]

    A unified approach to interpreting model predictions.Advances in neural information processing systems, 30, 2017

    Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions.Advances in neural information processing systems, 30, 2017

    2017

  19. [27]

    Lutz, Mimi P

    Chelsea S. Lutz, Mimi P. Huynh, Monica Schroeder, Sophia Anyatonwu, F. Scott Dahlgren, Gregory Danyluk, Danielle Fernandez, Sharon K. Greene, Nodar Kipshidze, Leann Liu, Osaro Mgbere, Lisa A. McHugh, Jennifer F. Myers, Alan Siniscalchi, Amy D. Sullivan, Nicole West, Michael A....

    2019

  20. [28]

    R package version 0.4.0, https://github.com/tidyverts/fable

    Mitchell O’Hara-Wild, Rob Hyndman, and Earo Wang.fable: Forecasting Models for Tidy Time Series, 2024. R package version 0.4.0, https://github.com/tidyverts/fable

    2024

  21. [29]

    Air: An r formatter and language server, 2025

    Posit Software, PBC. Air: An r formatter and language server, 2025. Version 0.8.1

    2025

  22. [30]

    Ray and Nicholas G

    Evan L. Ray and Nicholas G. Reich. Prediction of infectious disease epidemics via weighted density ensembles.PLOS Computational Biology, 14(2):e1005910, February 2018

    2018

  23. [31]

    R package version 0.1.0.9000, commit df75591bff3d68c3ecb1cad4798b1334230203f7

    Evan L Ray, Becky Sweger, and Lucie Contamin.hubExamples: Example Hub Data, 2025. R package version 0.1.0.9000, commit df75591bff3d68c3ecb1cad4798b1334230203f7

    2025

  24. [32]

    hubEvals: Basic tools for scoring hubverse forecasts, 2025

    Nicholas Reich, Evan Ray, Nikos Bosse, Matthew Cornell, Becky Sweger, and Kimberlyn Roosa. hubEvals: Basic tools for scoring hubverse forecasts, 2025. R package version 0.0.0.9001, commit 0329cb4f94f833c495bc3001cbacf82b103f35d2

    2025

  25. [33]

    Reich, Craig J

    Nicholas G. Reich, Craig J. McGowan, Teresa K. Yamana, Abhinav Tushar, Evan L. Ray, Dave Osthus, Sasikiran Kandula, Logan C. Brooks, Willow Crawford-Crudell, Graham Casey Gibson, Evan Moore, Rebecca Silva, Matthew Biggerstaff, Michael A. Johansson, Roni Rosenfeld, and Jeffrey ...

    2019

  26. [34]

    R package version 2.2.2

    Greg Ridgeway and GBM Developers.gbm: Generalized Boosted Regression Models, 2024. R package version 2.2.2

    2024

  27. [35]

    Multi-Model Ensembles in Infectious Disease and Public Health: Methods, Interpretation, and Implementation in R

    Li Shandross, Emily Howerton, Lucie Contamin, Harry Hochheiser, Anna Krystalli, Consortium of Infectious Disease Modeling Hubs, Nicholas G Reich, and Evan L Ray. Multi-Model Ensembles in Infectious Disease and Public Health: Methods, Interpretation, and Implementation in R. St...

    2026

  28. [36]

    R package version 1.0.0.9000, commit ca66c89340933124618aad8d100bfddd89637b7d

    Li Shandross, Emily Howerton, and Evan L Ray.hubEnsembles: Ensemble Meth- ods for Combining Hub Model Outputs, 2025. R package version 1.0.0.9000, commit ca66c89340933124618aad8d100bfddd89637b7d

    2025

  29. [37]

    A value for n-person games.Contribution to the Theory of Games, 2, 1953

    Lloyd S Shapley. A value for n-person games.Contribution to the Theory of Games, 2, 1953. 27

    1953

  30. [38]

    A new ensemble deep learning approach for exchange rates forecasting and trading.Advanced Engineering Informatics, 46:101160, 2020

    Shaolong Sun, Shouyang Wang, and Yunjie Wei. A new ensemble deep learning approach for exchange rates forecasting and trading.Advanced Engineering Informatics, 46:101160, 2020

    2020

  31. [39]

    Davis Vaughan and Matt Dancho.furrr: Apply Mapping Functions in Parallel using Futures,

  32. [40]

    R package version 0.3.1

  33. [41]

    The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt.Epidemics, 22:13–21, 2018

    C´ ecile Viboud, Kaiyuan Sun, Robert Gaffey, Marco Ajelli, Laura Fumanelli, Stefano Merler, Qian Zhang, Gerardo Chowell, Lone Simonsen, and Alessandro Vespignani. The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt.Epidemics, 22:13–21, 2018

    2018

  34. [42]

    testthat: Get started with testing.The R Journal, 3:5–10, 2011

    Hadley Wickham. testthat: Get started with testing.The R Journal, 3:5–10, 2011

    2011

  35. [43]

    Springer-Verlag New York, 2016

    Hadley Wickham.ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York, 2016

    2016

  36. [44]

    R package version 1.51

    Yihui Xie.knitr: A General-Purpose Package for Dynamic Report Generation in R, 2025. R package version 1.51

    2025

  37. [45]

    R package version 1.1.3

    Yachen Yan.MLmetrics: Machine Learning Evaluation Metrics, 2024. R package version 1.1.3

    2024

  38. [46]

    R package version 1.4.0

    Hao Zhu.kableExtra: Construct Complex Table with ’kable’ and Pipe Syntax, 2024. R package version 1.4.0. 28

    2024

Pith tools

Reviewed June 28, 2026 · model on record in the stance chip above.