{"id":"e29c5f93-8703-4cdb-831f-7a5666dc6eac","arxiv_id":"1908.05792","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"This paper presents multitask and transfer learning autotuning methods (MLA, TLA1, TLA2) for ScaLAPACK QR factorization and claims average runtime improvements that its own sections report inconsistently.","lead":"This paper proposes multitask and transfer learning methods to autotune the parameters of expensive high-performance computing applications, using ScaLAPACK's QR factorization as a test case. It reports that the multitask method MLA finds 1.5x faster runtimes on average than two standard autotuners, but the reported numbers are inconsistent and no code or data is provided.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's headline 1.5x/up-to-5x claim is directly contradicted by the Conclusion's 66%/40% numbers; without data/code the central experimental claim is internally inconsistent.","rationale":"The reader and I agree that the paper's empirical case is not reliable enough to accept. The reader's stated weakest assumption is the continuity/smoothness of the objective in task space (§3), which is a real and acknowledged limitation for transfer methods. However, I see the more decisive problem as an internal contradiction in the reported headline numbers: §6.2 gives 84%/94% win rates and up to 5x improvement, while the Conclusion gives 66% and up to 40%. Since the abstract's 1.5x claim is the central quantitative assertion, and since no code/data are provided to reconcile these summaries, the claim is internally inconsistent as written. This is not an ad hominem or a disagreement with a community consensus; it is a direct check of the paper's own evidence. A concrete reproduction with raw data would settle it. I therefore uphold the reader's REJECT verdict, while noting that the smoothness concern is secondary: even if smoothness holds for PDGEQRF, the numbers still conflict. The paper does contain useful ideas and clearly explains the methods, but the experimental basis for the central claim is not currently credible.","tokens_in":17507,"tokens_out":5281,"duration_ms":52566,"concrete_test":"Obtain the raw per-task best runtimes for the 50 tasks in Figure 1 and recompute both reported summaries: (i) the win rate of MLA versus OpenTuner and versus HpBandSter, and (ii) the maximum and average improvement ratio using the same aggregation as the paper. Then check whether the data reproduce not only the abstract's 1.5x average and 5x maximum but also the Conclusion's 66% win rate and 40% maximum. If the two sets of numbers cannot both be derived from one dataset, or if the 1.5x average changes when using median/geometric aggregation, the headline claim is not supported.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is empirical: MLA/TLA deliver a 1.5x average runtime improvement over OpenTuner and HpBandSter (§6.2, Figure 1). The paper's own summaries of the same experiment do not agree. §6.2 states that MLA wins in 42/50 (84%) cases against OpenTuner and 47/50 (94%) against HpBandSter, with 'average 1.5x (up to 5x) improvement.' The Conclusion states that 'MLA outperforms OpenTuner on 66% of the cases they are compared on, with an application runtime improvement of up to 40%.' These cannot both describe the same comparison: 84% ≠ 66% and 5x ≠ 40% improvement. Because the paper provides no code, no data, and no error bars, a reader cannot tell which number is correct, whether the 1.5x average was computed by arithmetic/geometric mean over tasks, or whether the discrepancy is a typo. The smoothness assumption in §3 is explicitly acknowledged and mainly affects TLA1/TLA2; the internal inconsistency attacks the MLA headline itself, so it is the more load-bearing concern.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multitask/transfer learning framework for autotuning expensive HPC applications. It defines a task space of problem characteristics and an input space of tuning parameters, builds a multi-output Gaussian process (the Linear Coregionalization Model) over a set of tasks (MLA), and then proposes two transfer methods: TLA1 predicts optimal parameter configurations for a new task without running the application, while TLA2 uses that prediction to center the initial sampling and incrementally extends the LCM to the new task. The evaluation tunes ScaLAPACK PDGEQRF on the Edison machine and compares against OpenTuner and HpBandSter. The paper claims an average 1.5x application-runtime improvement (up to 5x) for MLA, run-free TLA1 predictions competitive with those tuners, and argues that the methods are especially suitable for low-budget autotuning of expensive exascale applications.","tokens_in":17741,"tokens_out":6051,"duration_ms":57238,"significance":"If the empirical claims were fully supported, the paper would make a useful contribution: it gives a clean task-space formulation of autotuning, introduces a principled multi-output model (LCM) in this setting, and proposes a concrete low-cost transfer mechanism. The paper is also honest about its main modeling assumption, explicitly noting that the finite-dimensional task-space assumption may fail for problem classes such as sparse matrices in SuperLU. However, the contribution is primarily empirical, and the evidence as presented is not currently sufficient to support the headline numbers: the abstract and Section 6.2 report results that contradict the Conclusion, no code or data are provided, and the experiments lack error bars and significance tests. These issues bear directly on the paper's central claim rather than on presentation alone.","major_comments":[{"comment":"The central quantitative claim is internally inconsistent. Section 6.2 states that MLA leads to better application runtimes than OpenTuner in 42/50 (84%) cases and than HpBandSter in 47/50 (94%) cases, with an average 1.5x and up to 5x improvement, while Section 7 states that MLA outperforms OpenTuner on 66% of the cases with an application-runtime improvement of up to 40%. These cannot both describe the same MLA-vs-OpenTuner comparison. Because the paper provides neither the raw data nor the code, I cannot determine which number is correct, how the 'average' was computed, or whether the discrepancy is a typo. This must be resolved with a full, consistent report of the experimental results, including per-task data and a precise definition of the average improvement.","section":"§6.2 and §7"},{"comment":"The statistical support for the empirical claims is missing. Figure 1 compares 50 tasks and Figure 2 compares 10 tasks, but no error bars, confidence intervals, or significance tests are reported. The paper states that each application run was performed three times and the minimum was taken, but it does not report repeated tuning runs or variability across tasks. For example, the statement that TLA1 is 'competitive with OpenTuner' because the two methods 'outperform each other in 50% of the cases' rests on 10 tasks with no measure of uncertainty. The paper should report distributions or per-task tables and should state how many independent repetitions of the tuning process were performed.","section":"§6.2 and §6.3"},{"comment":"Several experimental details needed for reproducibility and interpretation are omitted. The seeds for the Latin Hypercube Sampling and Particle Swarm Optimization, the settings of the GPy optimization, the exact budget allocation in TLA2, and the handling of tied or invalid configurations are not specified. Section 6.2 says MLA costs less than OpenTuner in 24 cases and than HpBandSter in 38 cases, but then says the total runtime cost is similar to that of the other tuners; these statements need reconciliation. Section 6.3 states TLA2 outperforms OpenTuner in 7 of 10 cases and is outperformed in 2 of 10 cases without saying what happened in the remaining case. These omissions make the experiments difficult to reproduce and weaken the basis for the transfer-learning claims.","section":"§6.2–§6.4"},{"comment":"The transfer-learning claims depend on the smoothness/continuity assumption in task space, but the paper does not directly validate that assumption beyond the QR factorization case. The paper itself notes that the finite-dimensional task-space assumption may fail for problem classes such as sparse matrices in SuperLU. For the claims about TLA1 and TLA2 to be robust, the authors should provide a diagnostic test, such as comparing TLA1 predictions against independently tuned optima on held-out QR tasks, and should discuss how the methods are expected to degrade when the smoothness assumption fails.","section":"§3 and §5"}],"minor_comments":[{"comment":"The framework is referred to as 'HbBandSter' in Section 2.2 and 'HpBandSter' elsewhere; the reference [12] describes BOHB, so the naming should be made consistent and accurate throughout.","section":"§2.2"},{"comment":"The complexity statement 'Ø((tn)^3)' uses nonstandard notation; standard asymptotic notation such as O((t n)^3) or Θ((t n)^3) should be used.","section":"§6.2"},{"comment":"The comparison in Table 3 uses a single 500-by-500 matrix task; the text should explicitly acknowledge that this is anecdotal rather than general evidence for the low-budget advantage.","section":"Table 3"},{"comment":"The figure legend uses '.vs.' instead of 'vs.', and the caption should clarify which point markers correspond to OpenTuner and which to HpBandSter; this is a minor readability issue.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central empirical claim is currently unsupported by its own reporting: the abstract and Section 6.2 give different numbers from the Conclusion, and no code or data are provided. I would be willing to reconsider after a major revision that releases the data and code, reconciles the summary statistics, and adds proper error analysis. If such artifacts cannot be provided, the paper may need to be reframed as a methods proposal rather than an empirically validated contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is genuinely appealing: instead of tuning an application for each input problem in isolation, build a multi-output Gaussian process (LCM) that shares information across tasks, and then transfer that model to new problems via TLA1 (run-free prediction) and TLA2 (cheap centered sampling plus model update). The writing is clear, the mathematical setup is coherent, and the authors are honest about the key assumption—continuity of the objective in the task space—and even note where it fails (e.g., sparse matrix spaces in SuperLU). That is a thoughtful paper with a real contribution to autotuning for expensive HPC applications.\n\nThe problem is the experimental backbone. Section 6.2 says MLA wins against OpenTuner in 42/50 cases (84%) and against HpBandSter in 47/50 cases (94%), with an average 1.5x and up to 5x improvement. The Conclusion says MLA outperforms OpenTuner in 66% of cases with up to 40% improvement. Those are not two ways of saying the same thing—they are contradictory. Since no code or data are provided and no error bars are reported, a reader cannot tell which number is right, whether the average is arithmetic or geometric, or whether the discrepancy is a typo. That is a load-bearing flaw, not a minor one. The transfer learning experiments (TLA1/TLA2) are also reported without confidence intervals, and TLA2's centered sampling depends on the same smoothness assumption, so the empirical claims as a whole are not independently verifiable.\n\nThe comparison itself is defensible: MLA gets the same per-task budget of 20 evaluations, and the multitask sharing is the intended advantage, not a hidden trick. The paper also acknowledges MLA's higher tuning overhead, so the reader is not being misled there. But the contradictory headline numbers are exactly the kind of thing that makes a referee question whether the reported 1.5x would survive a scripted rerun.\n\nWho gets value from this? Researchers working on autotuning, especially for exascale systems, and methodologists interested in applying LCM to horizontal multitask optimization. The methodological novelty is real, and the limitations are openly discussed, which suggests the authors were not trying to hide anything. But the paper is not ready as is.\n\nMy recommendation: send it to peer review. A knowledgeable referee can assess the marginal value of the method and, more importantly, the authors should be asked to release code and data and reconcile the discrepancy. A desk reject would be too harsh for a paper this clearly presented and with this much potential. Just require the fix before acceptance.","headline":"A promising multitask/transfer autotuning framework whose reported gains are undermined by an internal inconsistency in the headline numbers; worth engaging but needs code/data and corrected results.","tokens_in":18314,"tokens_out":2121,"would_cite":false,"duration_ms":23278,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tuning a representative set of problems together, then transferring the learned model to new problems, gives on average 1.5x faster application runtime than existing tuners, with competitive zero-run predictions.","keywords":["multitask learning","transfer learning","autotuning","Bayesian optimization","multi-output Gaussian process","linear coregionalization model","ScaLAPACK","exascale applications"],"falsifier":"Take an application whose performance landscape is deliberately discontinuous in the task features, such as a sparse solver where adding or moving one non-zero entry flips the optimal block size, and measure whether TLA1's run-free predictions beat a fixed default configuration under the same budget; if they do not, the continuity premise is false for that class and the transfer gain disappears.","tokens_in":17295,"feed_emoji":"⚙️","tokens_out":6566,"duration_ms":61657,"temperature":0.7,"pith_summary":"This paper tries to show that autotuning—finding the parameter settings that make an expensive scientific application run fastest—should not be done one input problem at a time. Instead, tune a representative set of problems together using multitask learning, then transfer the resulting model to never-before-seen problems. On the ScaLAPACK QR factorization routine, the shared model (MLA) finds parameter configurations whose application runtime is on average 1.5x better than those found by the OpenTuner and HpBandSter baselines, and the transfer methods (TLA1 and TLA2) produce competitive configurations with zero or very few extra runs. That matters because exascale applications are so expensive to run that tuning budgets of only dozens of runs are realistic.","feed_headline":"Shared autotuning model beats one-at-a-time tuners by 1.5x","feed_subtitle":"A model built once on representative problems predicts good settings for new problems with zero extra runs.","key_machinery":"The load-bearing object is the Linear Coregionalization Model (LCM), a multi-output Gaussian process that expresses the runtime of each task as a linear combination of shared latent functions. Its covariance between two tasks is built as a Kronecker product sum $\\sum_q B_q \\otimes k_q(x,x')$ plus a diagonal noise term, where the matrices $B_q$ encode which tasks are related. This single covariance structure is what lets the model use data from all tuned tasks when predicting any one task, and it is reused in two ways: TLA1 trains one Gaussian process per tunable parameter to map task features directly to the predicted optimum configuration, while TLA2 samples new tasks in a normal distribution centered at that prediction and extends the existing model by updating only the added rows and columns of the $B_q$ matrices, reducing model update cost from cubic in the total sample count to cubic in the per-task sample count.","core_discovery":"On the paper's own terms, the central discovery is that performance knowledge is portable across input problems. The authors define a task space of problem features (matrix size, node and core counts) and an input space of tunable parameters, then model the objective function jointly over many tasks with a multi-output Gaussian process, the Linear Coregionalization Model (LCM). This lets each task's predictions borrow strength from related tasks. Using this model, MLA outperforms OpenTuner on 84% and HpBandSter on 94% of 50 test tasks with equal evaluation budgets, with an average 1.5x improvement in application runtime; TLA1 predicts the optimum of a new task without any runs at all, and TLA2 uses those predictions to center a small additional search, both remaining competitive with the baselines. The price is a higher one-time model-building cost, which the paper argues is offset in exascale settings because runs are extremely expensive and the model build can be parallelized.","pith_inferences":["An implicit consequence the paper does not state: the method turns autotuning into an amortized investment, and the accuracy of zero-run predictions should improve each time TLA2 tunes another task and updates the shared model, making the framework self-improving.","A testable extension would be to run MLA and TLA on a suite of unrelated kernels (dense LU, sparse iterative solvers, stencils) to see whether cross-task Gaussian process sharing still helps when performance landscapes are less smooth than PDGEQRF's.","The paper's own caveat about SuperLU suggests the finite-dimensional task-space assumption is the main boundary: for sparse matrices with no finite feature representation, the transfer methods should be expected to degrade, a case the authors leave open."],"forward_implications":["A single autotuning campaign on representative problems can replace repeated per-problem tuning, so later problems are handled by model transfer rather than new searches.","TLA1's zero-run predictions make tuning possible even when a single run of the application is too expensive to spend on search.","TLA2's update and downdate procedure means adding new tasks to an existing model is much cheaper than rebuilding it, so the shared model can grow as new problem instances arrive.","Because the gains come from shared structure, the same recipe should transfer to other applications whose problems admit a finite-dimensional feature description.","Under a tight evaluation budget, one-at-a-time tuners are less appropriate than transferred models, so low-budget autotuning should favor the multitask and transfer approach."],"supporting_citations":[{"why":"Supplies the multi-output Gaussian process and Linear Coregionalization Model formulation that MLA's shared model is built on.","marker":"[8]"},{"why":"OpenTuner is the state-of-the-art autotuner baseline that MLA and the transfer methods are compared against.","marker":"[17]"},{"why":"HpBandSter is the second state-of-the-art baseline, providing the comparative 1.5x improvement result.","marker":"[12]"},{"why":"The EGO algorithm is the Bayesian optimization loop that MLA adapts to the multi-task setting.","marker":"[19]"},{"why":"Provides the ScaLAPACK PDGEQRF routine used as the guiding application in all experiments.","marker":"[7]"},{"why":"GPy is the Gaussian process library used for the implementation and hyperparameter learning of the models.","marker":"[15]"},{"why":"Latin Hypercube Sampling is the space-filling sampling technique used to select representative tasks and initial configurations.","marker":"[24]"},{"why":"Sparse Cholesky update and downdate techniques are what make TLA2's cheap model extension possible in practice.","marker":"[10]"}],"fun_headline_variants":["Autotuning model reuses knowledge, beats baselines by 1.5x","Transfer learning predicts good settings without running app","Multitask autotuner cuts runtime 1.5x vs OpenTuner, HpBandSter","Shared knowledge autotuner: zero new runs for new tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole transfer step stands on the assumption that performance is continuous in the task space: similar input problems have similar optimal configurations, so what is learned on one set of problems can predict another.","fun_headline_variants_meta":{"raw":{"variants":["Autotuning model reuses knowledge, beats baselines by 1.5x","Transfer learning predicts good settings without running app","Multitask autotuner cuts runtime 1.5x vs OpenTuner, HpBandSter","Shared knowledge autotuner: zero new runs for new tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1332,"prompt_tokens":868,"completion_tokens":464,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":382}},"tokens_in":484,"tokens_out":464,"duration_ms":4770,"temperature":1.0,"reasoning_tokens":382,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:04:32.924925+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an application whose performance landscape is deliberately discontinuous in the task features, such as a sparse solver where adding or moving one non-zero entry flips the optimal block size, and measure whether TLA1's run-free predictions beat a fixed default configuration under the same budget; if they do not, the continuity premise is false for that class and the transfer gain disappears.","supporting_citations":[{"cited_title":"Chai and Christopher Williams","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-output Gaussian process and Linear Coregionalization Model formulation that MLA's shared model is built on."},{"cited_title":"OpenTuner: An Extensible Framework for Program Autotuning","cited_arxiv_id":null,"evidence_quote":"OpenTuner is the state-of-the-art autotuner baseline that MLA and the transfer methods are compared against."},{"cited_title":"Falkner, A","cited_arxiv_id":null,"evidence_quote":"HpBandSter is the second state-of-the-art baseline, providing the comparative 1.5x improvement result."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The EGO algorithm is the Bayesian optimization loop that MLA adapts to the multi-task setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ScaLAPACK PDGEQRF routine used as the guiding application in all experiments."},{"cited_title":"GPy: A gaussian process framework in python","cited_arxiv_id":null,"evidence_quote":"GPy is the Gaussian process library used for the implementation and hyperparameter learning of the models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Latin Hypercube Sampling is the space-filling sampling technique used to select representative tasks and initial configurations."},{"cited_title":"and Hager, William W","cited_arxiv_id":null,"evidence_quote":"Sparse Cholesky update and downdate techniques are what make TLA2's cheap model extension possible in practice."}],"review_version":1}