REVIEW 3 major objections 5 minor 31 references
A coarse-grained lookup table, read through Shapley-effect sensitivity and Pareto-front analysis, identifies which hyperparameters drive each objective and whether fine-tuning a model class is worthwhile at all.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
A framework using Shapley Effects and Pareto fronts ranks hyperparameter influence per objective from a coarse grid-search lookup table, without proposing a new optimizer.
T0 review reviewed 2026-08-01 challenge →
load-bearing objection A competent application of Shapley Effects to hyperparameter sensitivity, with the standard math done correctly, but the single-run lookup table leaves the rankings and Pareto fronts potentially noise-driven. the 3 major comments →
Which Hyperparameters Matter? A Game-Theoretic Framework for Interpretable Hyperparameter Sensitivity Analysis
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The paper's central claim: treating hyperparameters as players in a cooperative game turns sensitivity analysis into a problem a cheap grid can solve. A lookup table from one evaluation per coarse configuration is read through Sobol–Shapley effects, whose influence scores sum to unity per objective. A user-set threshold classifies each hyperparameter as influential or non-influential, and Pareto dominance identifies non-dominated configurations. Across three case studies the tools deliver distinct verdicts: the PINN spans roughly 0–99% accuracy with clear tuning headroom; the CIFAR-100 CNN saturates near 50% accuracy; the Adult-income DNN sits in a narrow 83–86% band. These conclusions, draw
What carries the argument
The load-bearing object is the Sobol–Shapley effect: the Shapley value of a hyperparameter when coalition values come from a first-order Sobol variance decomposition, v(S) = Var(E[Y | G_S]) / Var(Y). Conceptually, it splits each objective's output variance among hyperparameters, averaging marginal contributions over all coalitions so the scores sum to exactly 1 per objective. It is variance-based and model-agnostic — no gradients, and interactions and nonlinearities are captured — yet Shapley-based, so each player gets a unique attribution. Conditional variances are estimated by Monte Carlo sampling (N = 10^4) over conditional subsets of the lookup table, avoiding exhaustive retraining. The
Load-bearing premise
The load-bearing premise is that a lookup table built from exactly one training run per coarse-grained configuration reflects true hyperparameter influence well enough that the variance decompositions and Pareto rankings are signal rather than training noise — the paper acknowledges this noise in its methods section and its discussion of limitations, but does not quantify it or use repeated runs — and that the chosen search space and discretization are representative of the h
What would settle it
Run the coarse-grained grid multiple times with different random seeds — say five to ten repeats per configuration — and recompute the Shapley effects from each seed's lookup table. If the influential/non-influential classification flips across seeds, or if the between-seed variance of a hyperparameter's Shapley effect is comparable to its mean value, then the single-run lookup table is too noisy to support the sensitivity claims. A cheaper complementary check: fix a hyperparameter the framework labels non-influential, vary the rest, and see whether the objective distributions change materiall
If this is right
- Hyperparameters classified as non-influential can be fixed to a single representative value or restricted to a small candidate set, shrinking the search space before any fine-grained optimization begins.
- The Pareto front doubles as an early-stage model evaluation: it reveals whether a model class can reach acceptable performance for a task (as with the PINN) or has already saturated (as with the CIFAR-100 CNN), before expensive tuning is spent.
- Hyperparameter influence is application- and architecture-specific: the learning rate can dominate one model's accuracy and matter little in another, so sensitivity priorities must be derived per task rather than assumed from prior experience.
- Because Shapley effects sum to unity per objective, the framework yields a normalized comparison of influence across objectives, making trade-offs between accuracy and training cost explicit.
- The framework is not tied to coarse grid search: any input–output mapping of configurations to objectives can be fed through the same sensitivity and Pareto analysis, making it reusable as transferable prior knowledge for later hyperparameter optimization.
Where Pith is reading between the lines
- My inference: the framework is best read as a pre-screening stage for HPO. If the single-pass grid is reliable, the efficient strategy it implies is to spend the tuning budget only in the influential subspace — the paper suggests this direction but does not measure the computational savings or quality loss relative to full-space tuning.
- My inference: the sharpest test the paper does not run is a fix-and-refine experiment — freeze the non-influential hyperparameters, fine-tune only the influential ones, and compare the resulting objectives with full-space optimization. The gap between the two would quantify how much the sensitivity ranking is worth.
- My inference: because every configuration is evaluated once, the threshold τ is chosen without any estimate of run-to-run noise. A natural extension is repeated-seed evaluation that separates training stochasticity from hyperparameter-driven variance; that would make the influential/non-influential cut statistically principled rather than user-chosen.
- My inference: the transferability claim — that the sensitivity spectra constitute reusable prior knowledge for similar models and tasks — is asserted but untested. A direct check would compute Shapley spectra on one dataset or architecture and verify that the influential sets carry over to a neighboring task.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a game-theoretic framework for interpretable hyperparameter sensitivity analysis. Hyperparameters are treated as players in a cooperative game; a coarse-grained grid search builds a lookup table of configurations and multi-objective outputs, Shapley effects are used for global sensitivity analysis to rank hyperparameter influence per objective, and Pareto-front analysis identifies non-dominated configurations for early-stage model evaluation. The framework is demonstrated on three neural-network case studies: a PINN for a dual spring–mass–damper system, a CNN on CIFAR-100, and a DNN on the Adult Income dataset. The paper claims the resulting sensitivity rankings and Pareto fronts provide transferable prior knowledge and can guide subsequent optimization and reduce search space.
Significance. If the framework's claims were fully supported, it would provide a useful, model-agnostic interpretability tool for hyperparameter importance, building on established Shapley-effect methodology for global sensitivity analysis. The proposed pipeline is simple, computationally modest after the initial grid search, and the authors provide a public codebase. The case studies cover diverse architectures and objectives. However, the central empirical claim—that the rankings and Pareto fronts reliably reflect genuine hyperparameter influence—is currently not supported because the entire analysis rests on a single evaluation per configuration with no quantification of training stochasticity. This is a load-bearing gap that must be addressed before the practical recommendations can be accepted.
major comments (3)
- [§II-C, Eqs. (11)–(12)] The manuscript states that Pareto optimality is used to mitigate the effect of 'lucky' runs from single evaluations, but Pareto non-dominance is computed on the noisy objective vectors themselves. A configuration with a favorable noise realization is more likely to appear on the frontier, so the Pareto front does not filter noise; it is biased toward lucky runs. No repeated-seed validation is provided to show that the reported fronts are stable. This directly undermines the framework's practical claims of 'early-stage model evaluation' and the conclusions drawn from Fig. 4 (e.g., that the CNN has reached a saturated performance regime).
- [§II-B, Eqs. (3c), (5b)–(5c), (6)] All Shapley effects are computed from a lookup table where each configuration is evaluated once. The conditional-mean estimator bµ(S,g_S) in Eq. (3c) inherits a noise variance of roughly σ²/|I(S,g_S)|, which is larger for larger coalitions because fewer matching configurations exist. This noise propagates through every marginal contribution in Eq. (6) and into the reported Shapley rankings in Fig. 3. The paper acknowledges training stochasticity but never quantifies σ² or provides error bars, repeated-seed estimates, or an external check (e.g., fixing non-influential hyperparameters and showing performance is preserved). Without this, the influential/non-influential classification may be an artifact of seed variation.
- [§III-B (global sensitivity analysis) and §IV] The actionable output of the framework is the classification of hyperparameters as influential or non-influential via the threshold τ in Eq. (8). However, τ is chosen arbitrarily, no sensitivity analysis of the classification to τ is reported, and the classification inherits all the single-run noise described above. The conclusions that the DNN is 'comparatively less sensitive' and that fine-tuning would provide 'only limited improvement' are strong practical statements that rest entirely on unvalidated, noisy estimates. A minimal validation would be to rerun a subset of configurations with multiple seeds and show that the rankings and the conclusions in Fig. 3 are stable.
minor comments (5)
- [§II-B, Eq. (5b)] The unweighted average bµ_S over g_S is used in Eq. (5c). For the full-factorial designs considered here, an unweighted average happens to equal the overall mean when conditional means are exact, but this is not stated. Clarify the relationship to avoid confusion about weighted vs unweighted estimates.
- [§II-C] Please specify whether the random seed is fixed across all configurations or varies. If seeds vary, the 'lucky run' concern is as stated; if seeds are fixed, the noise source is different and should be described. This detail is essential for interpreting the single-run evaluations.
- [§III-B] Fig. 3 reports Shapley effects without any uncertainty indicators. Even a simple bootstrap over the lookup table entries would help assess stability. At minimum, the threshold τ used in each subplot should be stated in the caption or text.
- [General] The paper repeatedly refers to 'Sobol-Shapley' sensitivity; the method is a Shapley-effect decomposition of Sobol indices. Using the standard name 'Shapley effects' consistently would align with the cited literature.
- [Appendix A] The sample lookup tables are labeled 'Run 1', 'Run 2', etc., but the ordering across tables is inconsistent (e.g., PINN Run 1 is Tanh while CNN Run 1 is ReLU). This is fine, but a note on how runs are ordered would improve readability.
Circularity Check
No significant circularity; the Shapley/Pareto analysis is a standard descriptive decomposition of the generated lookup table, not a prediction forced by construction.
full rationale
The paper's derivation chain is self-contained in the mathematical sense. It first constructs a lookup table D by coarse-grained search (Eq. 1f), then applies standard Sobol–Shapley variance decomposition (Eqs. 3–6) and Pareto dominance (Eqs. 9–12) to that table. The Shapley effects are a parameter-free summary of the variance of D once the user-set threshold tau is chosen; no parameter is fitted to a subset of outcomes and then used to predict the same outcomes. The central 'findings'—e.g., that certain hyperparameters have low Shapley effect or that the Pareto front is concentrated in a low-accuracy region—are descriptive statements about D, not predictions of held-out data. The claim that these findings constitute 'transferable prior knowledge' for similar tasks is an untested inductive generalization, which is a validation/correctness limitation, not circularity. Section II-C's acknowledged single-evaluation noise is also a robustness concern: Pareto non-dominance computed on noisy objectives may not filter 'lucky' runs, but this does not make the analysis reduce to its own inputs. The only self-citation ([22], Stein and Singh) appears in a background list of GSA references and is not load-bearing for the framework's derivation. No circular step can be exhibited from the paper's equations or citations.
Axiom & Free-Parameter Ledger
free parameters (2)
- Threshold τ for influential/non-influential classification =
Not explicitly stated; user-defined per figure
- Coarse-grained search grids (candidate value sets per hyperparameter) =
e.g., learning rate {1e-3,1e-4}, layers {2,4}, batch size {16,64}, etc.
axioms (4)
- domain assumption The coarse-grained lookup table D is representative enough for global sensitivity analysis
- domain assumption Single evaluation per configuration is a valid proxy for expected performance
- standard math Sobol-Shapley properties (sum to unity, nonnegative marginal contributions) hold for the empirical estimator
- standard math Pareto dominance on sign-transformed objectives correctly orders configurations
Cite this review
Pith. "Pith review of Which Hyperparameters Matter? A Game-Theoretic Framework for Interpretable Hyperparameter Sensitivity Analysis." pith.science (2026). https://pith.science/paper/ICF2T5JH
@misc{pith2026260715884,
author = {Pith},
title = {Pith review of: Which Hyperparameters Matter? A Game-Theoretic Framework for Interpretable Hyperparameter Sensitivity Analysis},
year = {2026},
howpublished = {\url{https://pith.science/paper/ICF2T5JH}},
note = {Machine review of arXiv:2607.15884}
}
read the original abstract
This work presents a game-theoretic framework for interpretable hyperparameter-objective interaction analysis rather than proposing a new optimization algorithm. In the proposed framework, Shapley Effects are employed for global sensitivity analysis, while Pareto front sets are utilized to identify effective hyperparameter configurations and support early-stage model evaluation. The resulting analysis reveals which players (hyperparameters) are most influential with respect to different objectives in a given game (application). Consequently, the proposed framework provides interpretable insights into objective-aware hyperparameter interactions, enabling practitioners to guide subsequent optimization, reduce the search space, and perform early-stage model evaluation. The effectiveness of the proposed framework is demonstrated using three distinct neural network architectures across different problem domains under multi-objective settings.
Figures
Reference graph
Works this paper leans on
-
[1]
Hyperparameter Optimization: A Spectral Approach,
E. Hazan, A. Klivans, and Y . Yuan, “Hyperparameter Optimization: A Spectral Approach,” Jan. 2018, arXiv:1706.00764 [cs]
Pith/arXiv arXiv 2018
-
[2]
Comparative Analysis of Automated Machine Learning for Hyperparameter Opti- mization and Explainable Artificial Intelligence Models,
M. S. Khan, T. Peng, H. Akhlaq, and M. Adeel Khan, “Comparative Analysis of Automated Machine Learning for Hyperparameter Opti- mization and Explainable Artificial Intelligence Models,”IEEE Access, vol. 13, pp. 84 966–84 991, 2025
2025
-
[3]
Advanced hyperparameter optimization for improved spatial prediction of shallow landslides using extreme gradient boosting (XGBoost),
T. Kavzoglu and A. Teke, “Advanced hyperparameter optimization for improved spatial prediction of shallow landslides using extreme gradient boosting (XGBoost),”Bulletin of Engineering Geology and the Environment, vol. 81, no. 5, p. 201, Apr. 2022
2022
-
[4]
ShrinkHPO: Towards Ex- plainable Parallel Hyperparameter Optimization,
T. Mu, H. Wang, H. Tang, and X. Shao, “ShrinkHPO: Towards Ex- plainable Parallel Hyperparameter Optimization,” in2024 IEEE 40th International Conference on Data Engineering (ICDE), May 2024, pp. 4897–4910
2024
-
[5]
Hyperparame- ter Importance Analysis for Multi-Objective AutoML,
D. Theodorakopoulos, F. Stahl, and M. Lindauer, “Hyperparame- ter Importance Analysis for Multi-Objective AutoML,” Oct. 2024, arXiv:2405.07640 [cs]
Pith/arXiv arXiv 2024
-
[6]
Hyperparameter Optimization of Long Short Term Memory Models for Interpretable Electrical Fault Classification,
B. G. M. and G. N. Pillai, “Hyperparameter Optimization of Long Short Term Memory Models for Interpretable Electrical Fault Classification,” IEEE Access, vol. 11, pp. 123 688–123 704, 2023
2023
-
[7]
Explaining Bayesian Optimization byShapley Values Facilitates Human-AI Collabo- ration for Exosuit Personalization,
J. Rodemann, F. Croppi, P. Arens, Y . Sale, J. Herbinger, B. Bischl, E. Hllermeier, T. Augustin, C. J. Walsh, and G. Casalicchio, “Explaining Bayesian Optimization byShapley Values Facilitates Human-AI Collabo- ration for Exosuit Personalization,” inMachine Learning and Knowledge Discovery in Databases. Research Track and Applied Data Science Track, B. Pf...
2026
-
[8]
M. M. Islam, H. R. Rifat, M. S. B. Shahid, A. Akhter, M. A. Uddin, and K. M. M. Uddin, “Explainable Machine Learning for Efficient Diabetes Prediction Using Hyperparame- ter Tuning, SHAP Analysis, Partial Dependency, and LIME,”En- gineering Reports, vol. 7, no. 1, p. e13080, 2025, eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/eng2.13080
-
[9]
A Survey on Un- certainty Quantification Methods for Deep Learning,
W. He, Z. Jiang, T. Xiao, Z. Xu, and Y . Li, “A Survey on Un- certainty Quantification Methods for Deep Learning,” Jan. 2025, arXiv:2302.13425 [cs]
arXiv 2025
-
[10]
HyperSHAP: Shapley Values and Interactions for Explaining Hyperparameter Opti- mization,
M. Wever, M. Muschalik, F. Fumagalli, and M. Lindauer, “HyperSHAP: Shapley Values and Interactions for Explaining Hyperparameter Opti- mization,” Nov. 2025, arXiv:2502.01276 [cs]
arXiv 2025
-
[11]
Comparative SHAP Analysis on SVM and K-NN: Impacts of Hyperparameter Tuning on Model Explainability,
I. Boukrouh, F. Tayalati, and A. Azmani, “Comparative SHAP Analysis on SVM and K-NN: Impacts of Hyperparameter Tuning on Model Explainability,” in2024 IEEE International Conference on Artificial Intelligence in Engineering and Technology (IICAIET), Aug. 2024, pp. 194–198
2024
-
[12]
Solutions of Feature and Hyperparameter Model Selection in the Intelligent Manufacturing,
C.-Y . Wang, C.-Y . Huang, and Y .-H. Chiang, “Solutions of Feature and Hyperparameter Model Selection in the Intelligent Manufacturing,” Processes, vol. 10, no. 5, p. 862, May 2022
2022
-
[13]
Powershap: A Power-Full Shapley Feature Selection Method,
J. Verhaeghe, J. Van Der Donckt, F. Ongenae, and S. Van Hoecke, “Powershap: A Power-Full Shapley Feature Selection Method,” inMa- chine Learning and Knowledge Discovery in Databases, M.-R. Amini, S. Canu, A. Fischer, T. Guns, P. Kralj Novak, and G. Tsoumakas, Eds. Cham: Springer International Publishing, 2023, pp. 71–87
2023
-
[14]
17. A Value for n-Person Games,
L. S. Shapley, “17. A Value for n-Person Games,” inContributions to the Theory of Games (AM-28), Volume II, H. W. Kuhn and A. W. Tucker, Eds. Princeton University Press, Dec. 1953, pp. 307–318
1953
-
[15]
shapiq: Shapley Interactions for Machine Learning,
M. Muschalik, H. Baniecki, F. Fumagalli, P. Kolpaczki, B. Hammer, and E. Hllermeier, “shapiq: Shapley Interactions for Machine Learning,” Oct. 2024, arXiv:2410.01649 [cs]
Pith/arXiv arXiv 2024
-
[16]
Shapley Based Residual Decomposition for Instance Analysis,
T. Liu and A. S. Barnard, “Shapley Based Residual Decomposition for Instance Analysis,” inProceedings of the 40th International Conference on Machine Learning. PMLR, Jul. 2023, pp. 21 375–21 387
2023
-
[17]
Rethinking data shapley for data selection tasks: misleads and merits,
J. T. Wang, T. Yang, J. Zou, Y . Kwon, and R. Jia, “Rethinking data shapley for data selection tasks: misleads and merits,” inProceedings of the 41st International Conference on Machine Learning, ser. ICML’24, vol. 235. Vienna, Austria: JMLR.org, Jul. 2024, pp. 52 033–52 063
2024
-
[18]
On Shapley value for measuring importance of dependent inputs,
A. B. Owen and C. Prieur, “On Shapley value for measuring importance of dependent inputs,” Mar. 2017, arXiv:1610.02080 [math]
Pith/arXiv arXiv 2017
-
[19]
Shapley effects for sen- sitivity analysis with dependent inputs: bootstrap and kriging-based algorithms,
N. Benoumechiara and K. Elie-Dit-Cosaque, “Shapley effects for sen- sitivity analysis with dependent inputs: bootstrap and kriging-based algorithms,”ESAIM: Proceedings and Surveys, vol. 65, pp. 266–293, 2019
2019
-
[20]
B. Iooss and C. Prieur, “Shapley effects for sensitivity analysis with correlated inputs: comparisons with Sobol’ indices, numerical estimation and applications,” Nov. 2019, arXiv:1707.01334 [math]
Pith/arXiv arXiv 2019
-
[21]
Computing Shapley Effects for Sensitivity Analysis,
E. Plischke, G. Rabitti, and E. Borgonovo, “Computing Shapley Effects for Sensitivity Analysis,”SIAM/ASA Journal on Uncertainty Quantifi- cation, vol. 9, no. 4, pp. 1411–1437, Jan. 2021
2021
-
[22]
Shapley Effect Estimation using Polynomial Chaos,
A. Stein and T. Singh, “Shapley Effect Estimation using Polynomial Chaos,” Jan. 2023, arXiv:2301.04315 [stat]. [Online]. Available: http://arxiv.org/abs/2301.04315
Pith/arXiv arXiv 2023
-
[23]
L. Pheulpin, N. Bertrand, and V . Bacchi, “Uncertainty quantification and global sensitivity analysis with dependent inputs parameters: Application to a basic 2D-hydraulic model,”LHB, vol. 108, no. 1, p. 2015265, Dec. 2022, eprint: https://doi.org/10.1080/27678490.2021.2015265
arXiv 2022
-
[24]
A Review on Global Sensitivity Analysis Methods,
B. Iooss and P. Lematre, “A Review on Global Sensitivity Analysis Methods,” inUncertainty Management in Simulation-Optimization of Complex Systems: Algorithms and Applications, G. Dellino and C. Mel- oni, Eds. Boston, MA: Springer US, 2015, pp. 101–122
2015
-
[25]
Efficient computation of global sensitivity indices using sparse polynomial chaos expansions,
G. Blatman and B. Sudret, “Efficient computation of global sensitivity indices using sparse polynomial chaos expansions,”Reliability Engineer- ing & System Safety, vol. 95, no. 11, pp. 1216–1229, Nov. 2010
2010
-
[26]
Derivative-based Shapley value for global sensitivity analysis and machine learning explainability,
H. Duan and G. kten, “Derivative-based Shapley value for global sensitivity analysis and machine learning explainability,” Mar. 2023, arXiv:2303.15183 [cs]
Pith/arXiv arXiv 2023
-
[27]
Multi-objective optimisation using evolutionary algorithms: an introduction,
K. Deb, “Multi-objective optimisation using evolutionary algorithms: an introduction,” inMulti-objective evolutionary optimisation for product design and manufacturing. Springer, 2011, pp. 3–34
2011
-
[28]
Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations,
M. Raissi, P. Perdikaris, and G. Karniadakis, “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations,” Journal of Computational Physics, vol. 378, pp. 686–707, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0021999118307125
2019
-
[29]
Learning multiple layers of features from tiny images,
A. Krizhevsky, “Learning multiple layers of features from tiny images,”
-
[30]
B. Becker and R. Kohavi, “Adult Dataset,” https://archive.ics.uci.edu/ml/datasets/Adult, 1996, 10.24432/C5XW20. [Online]. Available: https://archive.ics.uci.edu/ml/datasets/Adult APPENDIX A. Coarse-Grained Search Results for Each Model The evaluated objectives (y 1, y2, . . .) and their corresponding players (p1, p2, . . .) from grid search are stored in ...
doi:10.24432/c5xw20 1996
-
[2009]
Available: https://api.semanticscholar.org/CorpusID: 18268744
[Online]. Available: https://api.semanticscholar.org/CorpusID: 18268744
This paper was first reviewed by deepseek-v4-flash on August 1, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.