REVIEW 4 major objections 4 minor 1 cited by
Multitask and Transfer Learning for Autotuning Exascale Applications
T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Tuning a representative set of problems together, then transferring the learned model to new problems, gives on average 1.5x faster application runtime than existing tuners, with competitive zero-run predictions.
desk verdict A promising multitask/transfer autotuning framework whose reported gains are undermined by an internal inconsistency in the headline numbers; worth engaging but needs code/data and corrected results. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Linear Coregionalization Model (LCM), a multi-output Gaussian process that expresses the runtime of each task as a linear combination of shared latent functions. Its covariance between two tasks is built as a Kronecker product sum $\sum_q B_q \otimes k_q(x,x')$ plus a diagonal noise term, where the matrices $B_q$ encode which tasks are related. This single covariance structure is what lets the model use data from all tuned tasks when predicting any one task, and it is reused in two ways: TLA1 trains one Gaussian process per tunable parameter to map task features directly to the predicted optimum configuration, while TLA2 samples new tasks in a normal distribution centered at that prediction and extends the existing model by updating only the added rows and columns of the $B_q$ matrices, reducing model update cost from cubic in the total sample count to cubic in the per-task sample count.
What would settle it
Take an application whose performance landscape is deliberately discontinuous in the task features, such as a sparse solver where adding or moving one non-zero entry flips the optimal block size, and measure whether TLA1's run-free predictions beat a fixed default configuration under the same budget; if they do not, the continuity premise is false for that class and the transfer gain disappears.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that performance knowledge is portable across input problems. The authors define a task space of problem features (matrix size, node and core counts) and an input space of tunable parameters, then model the objective function jointly over many tasks with a multi-output Gaussian process, the Linear Coregionalization Model (LCM). This lets each task's predictions borrow strength from related tasks. Using this model, MLA outperforms OpenTuner on 84% and HpBandSter on 94% of 50 test tasks with equal evaluation budgets, with an average 1.5x improvement in application runtime; TLA1 predicts the optimum of a new task without any runs at all, and TLA2 uses those predictions to center a small additional search, both remaining competitive with the baselines. The price is a higher one-time model-building cost, which the paper argues is offset in exascale settings because runs are extremely expensive and the model build can be parallelized.
Load-bearing premise
The whole transfer step stands on the assumption that performance is continuous in the task space: similar input problems have similar optimal configurations, so what is learned on one set of problems can predict another.
Editorial extensions
If this is right
- A single autotuning campaign on representative problems can replace repeated per-problem tuning, so later problems are handled by model transfer rather than new searches.
- TLA1's zero-run predictions make tuning possible even when a single run of the application is too expensive to spend on search.
- TLA2's update and downdate procedure means adding new tasks to an existing model is much cheaper than rebuilding it, so the shared model can grow as new problem instances arrive.
- Because the gains come from shared structure, the same recipe should transfer to other applications whose problems admit a finite-dimensional feature description.
- Under a tight evaluation budget, one-at-a-time tuners are less appropriate than transferred models, so low-budget autotuning should favor the multitask and transfer approach.
Reading between the lines
- An implicit consequence the paper does not state: the method turns autotuning into an amortized investment, and the accuracy of zero-run predictions should improve each time TLA2 tunes another task and updates the shared model, making the framework self-improving.
- A testable extension would be to run MLA and TLA on a suite of unrelated kernels (dense LU, sparse iterative solvers, stencils) to see whether cross-task Gaussian process sharing still helps when performance landscapes are less smooth than PDGEQRF's.
- The paper's own caveat about SuperLU suggests the finite-dimensional task-space assumption is the main boundary: for sparse matrices with no finite feature representation, the transfer methods should be expected to degrade, a case the authors leave open.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a multitask/transfer learning framework for autotuning expensive HPC applications. It defines a task space of problem characteristics and an input space of tuning parameters, builds a multi-output Gaussian process (the Linear Coregionalization Model) over a set of tasks (MLA), and then proposes two transfer methods: TLA1 predicts optimal parameter configurations for a new task without running the application, while TLA2 uses that prediction to center the initial sampling and incrementally extends the LCM to the new task. The evaluation tunes ScaLAPACK PDGEQRF on the Edison machine and compares against OpenTuner and HpBandSter. The paper claims an average 1.5x application-runtime improvement (up to 5x) for MLA, run-free TLA1 predictions competitive with those tuners, and argues that the methods are especially suitable for low-budget autotuning of expensive exascale applications.
Significance. If the empirical claims were fully supported, the paper would make a useful contribution: it gives a clean task-space formulation of autotuning, introduces a principled multi-output model (LCM) in this setting, and proposes a concrete low-cost transfer mechanism. The paper is also honest about its main modeling assumption, explicitly noting that the finite-dimensional task-space assumption may fail for problem classes such as sparse matrices in SuperLU. However, the contribution is primarily empirical, and the evidence as presented is not currently sufficient to support the headline numbers: the abstract and Section 6.2 report results that contradict the Conclusion, no code or data are provided, and the experiments lack error bars and significance tests. These issues bear directly on the paper's central claim rather than on presentation alone.
major comments (4)
- [§6.2 and §7] The central quantitative claim is internally inconsistent. Section 6.2 states that MLA leads to better application runtimes than OpenTuner in 42/50 (84%) cases and than HpBandSter in 47/50 (94%) cases, with an average 1.5x and up to 5x improvement, while Section 7 states that MLA outperforms OpenTuner on 66% of the cases with an application-runtime improvement of up to 40%. These cannot both describe the same MLA-vs-OpenTuner comparison. Because the paper provides neither the raw data nor the code, I cannot determine which number is correct, how the 'average' was computed, or whether the discrepancy is a typo. This must be resolved with a full, consistent report of the experimental results, including per-task data and a precise definition of the average improvement.
- [§6.2 and §6.3] The statistical support for the empirical claims is missing. Figure 1 compares 50 tasks and Figure 2 compares 10 tasks, but no error bars, confidence intervals, or significance tests are reported. The paper states that each application run was performed three times and the minimum was taken, but it does not report repeated tuning runs or variability across tasks. For example, the statement that TLA1 is 'competitive with OpenTuner' because the two methods 'outperform each other in 50% of the cases' rests on 10 tasks with no measure of uncertainty. The paper should report distributions or per-task tables and should state how many independent repetitions of the tuning process were performed.
- [§6.2–§6.4] Several experimental details needed for reproducibility and interpretation are omitted. The seeds for the Latin Hypercube Sampling and Particle Swarm Optimization, the settings of the GPy optimization, the exact budget allocation in TLA2, and the handling of tied or invalid configurations are not specified. Section 6.2 says MLA costs less than OpenTuner in 24 cases and than HpBandSter in 38 cases, but then says the total runtime cost is similar to that of the other tuners; these statements need reconciliation. Section 6.3 states TLA2 outperforms OpenTuner in 7 of 10 cases and is outperformed in 2 of 10 cases without saying what happened in the remaining case. These omissions make the experiments difficult to reproduce and weaken the basis for the transfer-learning claims.
- [§3 and §5] The transfer-learning claims depend on the smoothness/continuity assumption in task space, but the paper does not directly validate that assumption beyond the QR factorization case. The paper itself notes that the finite-dimensional task-space assumption may fail for problem classes such as sparse matrices in SuperLU. For the claims about TLA1 and TLA2 to be robust, the authors should provide a diagnostic test, such as comparing TLA1 predictions against independently tuned optima on held-out QR tasks, and should discuss how the methods are expected to degrade when the smoothness assumption fails.
minor comments (4)
- [§2.2] The framework is referred to as 'HbBandSter' in Section 2.2 and 'HpBandSter' elsewhere; the reference [12] describes BOHB, so the naming should be made consistent and accurate throughout.
- [§6.2] The complexity statement 'Ø((tn)^3)' uses nonstandard notation; standard asymptotic notation such as O((t n)^3) or Θ((t n)^3) should be used.
- [Table 3] The comparison in Table 3 uses a single 500-by-500 matrix task; the text should explicitly acknowledge that this is anecdotal rather than general evidence for the low-budget advantage.
- [Figure 1] The figure legend uses '.vs.' instead of 'vs.', and the caption should clarify which point markers correspond to OpenTuner and which to HpBandSter; this is a minor readability issue.
Circularity Check
No significant circularity: the paper's claims are empirical comparisons against external tuners, and its learned models are standard fitting procedures evaluated on held-out tasks.
full rationale
The paper's central claims are experimental: MLA and TLA methods are compared with OpenTuner and HpBandSter on runtime data, and the results are reported as win rates and speedup ratios. MLA fits a multi-output Gaussian process (LCM) to runtime samples, which is standard supervised fitting, not a derivation of the target quantity from an assumption that contains it. TLA1 builds separate Gaussian processes on the optima found by MLA and predicts configurations for 10 newly generated tasks; those tasks are not the ones used to fit the model, so the evaluation is out-of-sample. TLA2 uses the TLA1 prediction only as the center of its initial sampling distribution, while still spending a budget of 100 objective-function evaluations; the claim that TLA2 outperforms TLA1 in 6 of 10 cases is an empirical result that could have come out differently and is not forced by construction. The smoothness/continuity assumption in Section 3 is explicitly stated as an assumption, not used to derive the measured comparisons. The paper does not rely on a load-bearing self-citation chain: citations to GPy, LCM, EGO, OpenTuner, and HpBandSter are standard external references, and the authors' own prior work appears only as background (e.g., ScaLAPACK, SuperLU). The internal inconsistency between Section 6.2 ('42 (84%) and 47 (94%) cases' with 'average 1.5x (up to 5x) improvement') and the Conclusion ('66% of the cases' with 'up to 40%' improvement) is a numerical consistency problem in reporting, not a circularity of reasoning, because neither number is an input to the other in a derivational sense. No step in the paper reduces, by its own equations or by self-citation, to its own inputs.
Assumptions & free parameters
free parameters (4)
- Q (number of latent functions in LCM) =
20
- Number of inducing points for sparse GP approximation =
6*sqrt(t*n)
- Initial sample split in MLA =
12 initial samples, 8 optimization samples per task
- LCM kernel hyperparameters =
Learned via log-likelihood optimization in GPy
assumptions (5)
- domain assumption The objective function is continuous and similar for similar tasks in the task space.
- domain assumption The task space can be characterized by a finite-dimensional vector (m, n, nodes, cores) for PDGEQRF.
- domain assumption LCM latent functions are independent and the covariance between outputs is a linear combination of their covariances.
- domain assumption Latin hypercube sampling gives a representative set of tasks and initial parameter configurations.
- domain assumption Parameter constraints can be reformulated into a valid parameter space without changing the location of optima.
Cite this review
Pith. "Pith review of Multitask and Transfer Learning for Autotuning Exascale Applications." pith.science (2026). https://pith.science/paper/JVVNRLJE
@misc{pith2026190805792,
author = {Pith},
title = {Pith review of: Multitask and Transfer Learning for Autotuning Exascale Applications},
year = {2026},
howpublished = {\url{https://pith.science/paper/JVVNRLJE}},
note = {Machine review of arXiv:1908.05792}
}
abstract
Multitask learning and transfer learning have proven to be useful in the field of machine learning when additional knowledge is available to help a prediction task. We aim at deriving methods following these paradigms for use in autotuning, where the goal is to find the optimal performance parameters of an application treated as a black-box function. We show comparative results with state-of-the-art autotuning techniques. For instance, we observe an average $1.5x$ improvement of the application runtime compared to the OpenTuner and HpBandSter autotuners. We explain how our approaches can be more suitable than some state-of-the-art autotuners for the tuning of any application in general and of expensive exascale applications in particular.
Figures
Forward citations
Cited by 1 Pith paper
-
HPC Application Parameter Autotuning on Edge Devices: A Bandit Learning Approach
A lightweight UCB-based autotuner (LASP) tuned four HPC applications on a Jetson Nano, finding configurations within 12% of the oracle and improving over defaults by 6-14%.
Reference graph
Works this paper leans on
-
[1]
Journel, A. G. and Huijbregts, C. J. . Mining geostatistics / [by] A. G. Journel and Ch. J. Huijbregts . Academic Press London ; New York , 1978
work page 1978
-
[2]
A. Abdelfattah, A. Haidar, S. Tomov, and J. J. Dongarra. Fast cholesky factorization on gpus for batch and native modes in MAGMA. J. Comput. Science , 20:85–93, 2017
work page 2017
-
[3]
Multi-fidelity optimization via surrogate modelling
Alexander I.J Forrester and Andr´ as S´ obester and Andy J Keane. Multi-fidelity optimization via surrogate modelling. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences , 463(2088):3251–3269, 2007
work page 2007
-
[4]
H. Anzt, B. Haugen, J. Kurzak, P. Luszczek, and J. Dongarra. Experiences in autotuning matrix multiplication for energy minimization on gpus. Concurrency and Computation: Practice and Experience , 27:5096 – 5113, Oct-12-2015 2015
work page 2015
-
[5]
B. Shahriari and K. Swersky and Z. Wang and R. P. Adams and N. de Freitas. Taking the Human Out of the Loop: A Review of Bayesian Optimization. Proceedings of the IEEE, 104(1):148–175, Jan 2016
work page 2016
-
[6]
R. Bellman. Dynamic Programming. Princeton University Press, Princeton, NJ, USA, 1 edition, 1957
1957
-
[7]
Blackford, L. S. and Choi, J. and Cleary, A. and D’Azevedo, E. and Demmel, J. and Dhillon, I. and Dongarra, J. and Hammarling, S. and Henry, G. and Petitet, A. and Stanley, K. and Walker, D. and Whaley, R. C. ScaLAPACK Users’ Guide. Society for Industrial and Applied Mathematics, Philadelphia, PA, 1997
work page 1997
-
[8]
Bonilla, Edwin V and Kian M. Chai and Christopher Williams. Multi-task Gaussian Process Prediction. In J. C. Platt and D. Koller and Y. Singer and S. T. Roweis, editor, Advances in Neural Information Processing Systems 20 , pages 153–160. Curran Associates, Inc., 2008. 16
work page 2008
Show all 34 references
-
[9]
and Larsen, Kasper Green and P˘ atra¸ scu, Mihai
Chan, Timothy M. and Larsen, Kasper Green and P˘ atra¸ scu, Mihai. Orthogonal Range Searching on the RAM, Revisited. In Proceedings of the Twenty-seventh Annual Symposium on Computational Geometry , SoCG ’11, pages 1–10, New York, NY, USA, 2011. ACM
2011
-
[10]
and Hager, William W
Davis, Timothy A. and Hager, William W. Dynamic Supernodes in Sparse Cholesky Update/Downdate and Triangular Solves. ACM Trans. Math. Softw. , 35(4):27:1–27:23, feb 2009
2009
-
[11]
Communication-optimal parallel and se- quential QR and LU factorizations
Demmel, James and Grigori, Laura and Hoemmen, Mark and Langou, Julien. Communication-optimal parallel and se- quential QR and LU factorizations. SIAM Journal on Scientific Computing , 34(1):A206–A239, 2012
2012
-
[12]
Falkner, A
S. Falkner, A. Klein, and F. Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research, pages 1437–1446, S...
2018
-
[13]
The design of experiments
Fisher, R.A. The design of experiments. 1935 . Oliver and Boyd, Edinburgh, 1935
1935
-
[14]
Geostatistics for natural resources evaluation
Goovaerts, Pierre. Geostatistics for natural resources evaluation . Oxford University Press, Oxford, 1997
1997
-
[15]
GPy: A gaussian process framework in python
GPy. GPy: A gaussian process framework in python. http://github.com/SheffieldML/GPy, since 2012
2012
-
[16]
Automatic Relevance Determination (ARD) , pages 221–227
Husmeier, Dirk. Automatic Relevance Determination (ARD) , pages 221–227. Springer London, London, 1999
1999
-
[17]
OpenTuner: An Extensible Framework for Program Autotuning
Jason Ansel and Shoaib Kamil and Kalyan Veeramachaneni and Jonathan Ragan-Kelley and Jeffrey Bosboom and Una- May O’Reilly and Saman Amarasinghe. OpenTuner: An Extensible Framework for Program Autotuning. In International Conference on Parallel Architectures and Compilation Tec...
2014
-
[18]
Nelder and Roger Mead
John A. Nelder and Roger Mead. A simplex method for function minimization. Computer Journal, 7:308–313, 1965
1965
-
[19]
D. R. Jones, M. Schonlau, and W. J. Welch. Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13(4):455–492, Dec 1998
1998
-
[20]
and Veinott,Jr., Arthur F
Katehakis, Michael N. and Veinott,Jr., Arthur F. The Multi-armed Bandit Problem: Decomposition and Computation. Mathematics of Operations Research, 12(2):262–268, May 1987
1987
-
[21]
Private Communication, 2018
Laura Grigori. Private Communication, 2018
2018
-
[22]
X. S. Li and J. W. Demmel. SuperLU DIST: A scalable distributed-memory sparse direct solver for unsymmetric linear systems. ACM Trans. Math. Softw. , 29(2):110–140, June 2003
2003
-
[23]
Hyperband: A Novel Bandit-based Approach to Hyperparameter Optimization
Li, Lisha and Jamieson, Kevin and DeSalvo, Giulia and Rostamizadeh, Afshin and Talwalkar, Ameet. Hyperband: A Novel Bandit-based Approach to Hyperparameter Optimization. J. Mach. Learn. Res. , 18(1):6765–6816, Jan. 2017
2017
-
[24]
M. D. McKay and R. J. Beckman and W. J. Conover. A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code. Technometrics, 21(2):239–245, 1979
1979
-
[25]
Particle Swarm Optimization: Theory, Techniques and Applications
Olsson, Andrea E. Particle Swarm Optimization: Theory, Techniques and Applications . Nova Science Publishers, Inc., Commack, NY, USA, 2010
2010
-
[26]
Balaprakash
P. Balaprakash. SuRF: Search using Random Forest, 2015. SuRF is a model-based search module for automatic performance tuning. It adopts random forest supervised learning algorithm for modeling the performances as a function of input parameters within the search. SuRF samples a...
2015
-
[27]
Balaprakash and J
P. Balaprakash and J. Dongarra and T. Gamblin and M. Hall and J. K. Hollingsworth and B. Norris and R. Vuduc. Autotuning in High- Performance Computing Applications. Proceedings of the IEEE, ():1–16, 2018
2018
-
[28]
Rasmussen, Carl Edward and Williams, Christopher K. I. Gaussian Processes for Machine Learning (Adaptive Computa- tion and Machine Learning) . The MIT Press, 2005
2005
-
[29]
S. J. Pan and Q. Yang. A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering , 22(10):1345–1359, Oct 2010
2010
-
[30]
Kirkpatrick and C
S. Kirkpatrick and C. D. Gelatt and M. P. Vecchi. Optimization by simulated annealing. SCIENCE, 220(4598):671–680, 1983
1983
-
[31]
Snoek, H
J. Snoek, H. Larochelle, and R. P. Adams. Practical bayesian optimization of machine learning algorithms. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25 , pages 2951–2959. Curran Associates, Inc., 2012
2012
-
[32]
and Patnaik, Lalit M
Srinivas, M. and Patnaik, Lalit M. Genetic Algorithms: A Survey. Computer, 27(6):17–26, June 1994
1994
-
[33]
Xiaoye S. Li. An overview of SuperLU: Algorithms, implementation, and user interface. ACM Trans. Math. Softw. , 31(3):302–325, 2005
2005
-
[34]
A Survey on Multi-Task Learning
Yu Zhang and Qiang Yang. A Survey on Multi-Task Learning. CoRR, abs/1707.08114, 2017. 17
2017 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.