{"id":"72321d74-c91f-4d62-9c73-c3097c52f9f7","arxiv_id":"2607.25831","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"WarmTuner learns a source-code-conditioned flag policy offline and refines it online with GRPO, reporting 1.732x average speedup over GCC -O3 but not a significant gain over GroupTuner.","lead":"WarmTuner trains a program-specific policy on historical compiler tuning records, then refines it on each new program using measured speedups. It reports an average 1.732x speedup over GCC -O3 across 30 benchmark programs, though the gain over the strongest baseline is not statistically significant.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GRPO group size K=8 was selected using the evaluation programs (Fig. 2), so Table II's speedups are test-set-informed, not unbiased; the 1.732x claim may be inflated.","rationale":"The central claim is empirical: WarmTuner's 1.732x average speedup and 14/30 best results. The reader's weakest assumption concerned the offline RIO search records. I find that partially mitigated: the WarmTuner_PDCAT ablation (Section IV-B4) shows the framework works with records from a different tuner, and the NoPre ablation shows the offline prior helps but the online GRPO stage also contributes. The more load-bearing flaw is that K was selected after observing performance on the evaluation programs. The footnote in Section IV-B confirms the ablation used one split's training/testing programs; Figure 2 is then used to justify K=8. This is classic test-set leakage and directly undermines the numerical claims. The reader flagged this in the rationale but did not make it the weakest assumption. Because the paper can likely be fixed by re-running with a pre-specified or validation-selected K, the appropriate verdict remains CONDITIONAL rather than REJECT. I therefore keep the reader's conditional verdict while shifting the primary justification.","tokens_in":19825,"tokens_out":8632,"duration_ms":76803,"concrete_test":"Re-run the full RQ1 protocol with K fixed to 16 (a pre-specified value not informed by evaluation results), and recompute the 5,000s average speedup, best/worst counts, and paired t-test versus GroupTuner. If the average speedup falls below GroupTuner's 1.669x or the p-value remains >0.05, the headline superiority claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is test-set leakage in hyperparameter selection. Section IV-B5 compares K=4, K=8, and K=16 on the testing programs used in RQ1 (footnote: 'we use one selection result as initial tuning and testing programs'), and then fixes K=8 as the default for the main experiments. This means the evaluation programs were used to select a hyperparameter, so the reported 1.732x average speedup and 14/30 best results are not independent estimates of WarmTuner's performance on unseen programs. The paper even reports p=0.174 against GroupTuner, so any change in model selection could alter the qualitative conclusion. This concern is distinct from the offline-data-source assumption: even if RIO records contain sufficient signal, the quantitative claims are still compromised because the test set informed the algorithm configuration.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"WarmTuner proposes an offline-to-online reinforcement learning framework for GCC flag selection. In the offline stage, a program-conditioned Bernoulli policy over 126 optimization flags is pretrained by per-flag binary cross-entropy on historical good configurations collected by RIO, with programs represented by averaged CodeBERT chunk embeddings. In the online stage, the same policy is refined on the target program using Group Relative Policy Optimization (GRPO), where the reward is the measured speedup over GCC -O3 and candidates are compared within a group of K=8. The paper evaluates WarmTuner on cBench and PolyBench under 1,250/2,500/5,000-second budgets, compares it with RIO, SRTuner, GroupTuner, and PDCAT, and reports a 1.732x average speedup over -O3 at 5,000 seconds, with 14/30 best results, plus ablations of offline pretraining, program representation, optimizer choice, data source, and group size.","tokens_in":20042,"tokens_out":7737,"duration_ms":76901,"significance":"If the results are clean, the paper makes a useful contribution: it directly addresses a real limitation of existing compiler autotuning priors—namely, that they are fixed once online search begins—by making the prior program-conditioned and online-adaptable. The GRPO instantiation is a sensible fit to the group-evaluation structure and avoids a separate value model. The measurement protocol is carefully described, and the availability of a public artifact is a strength. However, the methodological and statistical concerns below must be resolved before the headline claims can be accepted at face value.","major_comments":[{"comment":"The group size K=8 used in RQ1 is selected/validated on the same testing programs used in RQ1. Footnote 1 states that the ablation uses one selection result as initial tuning and testing programs, and §IV-B5 says the variants are compared 'keeping the same testing programs.' As written, this means the Table II results are not independent held-out estimates: test data informed a hyperparameter. Please either state clearly that K=8 was fixed a priori and that Figure 2 is a post hoc confirmation, or re-run with a proper validation split. The issue matters because Figure 2 shows K=16 yields higher average speedups at every budget, so the choice of K materially affects the quantitative comparison, and the p=0.174 against GroupTuner leaves little margin.","section":"§IV-B5, footnote 1, Figure 2"},{"comment":"The abstract and Section I claim WarmTuner 'significantly outperforms the compared techniques.' But §IV-A1 reports p=1.74e-1 against GroupTuner, the strongest baseline, i.e., the improvement is not statistically significant. The significant comparisons are only against RIO, SRTuner, and PDCAT. The claim should be qualified in the abstract and conclusion to avoid overstating the result; as written, the central contribution statement is not supported by the reported statistics.","section":"Abstract; §IV-A1"},{"comment":"Table II reports only point estimates (medians/averages) with no variance, confidence intervals, or per-repeat/per-split distributions, even though the protocol repeats each technique five times and uses three train/test splits. The paired t-test p-values are useful but insufficient to establish that WarmTuner 'performs consistently' when the margin over GroupTuner is small (1.732 vs. 1.669). Please report dispersion measures (e.g., standard deviations or confidence intervals over the five repeats and the three splits), especially for the close GroupTuner comparison.","section":"§III-B, §IV-A1, Table II"}],"minor_comments":[{"comment":"Several entries lack spacing between consecutive numbers (e.g., '1.3361.392', '1.4581.494', '1.3831.416'). Please fix the formatting and consider splitting the table for readability.","section":"Table II"},{"comment":"The text says K=16 gives 'comparable performance at 1,250 seconds and only slightly higher speedups under larger budgets,' but the plotted values appear to show K=16 exceeding K=8 by roughly 0.06 at every budget. Please reconcile the description with the figure.","section":"§IV-B5, Figure 2"},{"comment":"The dataset description says cBench has 20 programs and Table II lists 20 cBench IDs, but Table I appears to label only C1–C20 while the text says 'P1–P30 denote the 30 PolyBench programs.' Clarify how the 30 testing programs in Table II were selected from the 50 total programs and how per-program averages across appearances are computed.","section":"§III-A, Table I"},{"comment":"The paper states that offline pretraining costs about 9.18 minutes per split and calls this negligible, but it does not include this cost in the comparison budgets. If the claim is that WarmTuner is practical under the stated budgets, please state explicitly whether the offline cost is excluded from the 1,250/2,500/5,000-second budgets.","section":"§IV-A1"}],"recommendation":"major_revision","confidential_remarks":"The offline-to-online idea is publishable and the artifact is a plus. The main risk is the K-selection procedure using test programs, which the authors should fix or clarify, and the overstatement of significance against GroupTuner. If the GroupTuner comparison cannot be improved, the paper should be repositioned as showing significant gains over three baselines and a competitive but not significant edge over GroupTuner."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: WarmTuner is a real but incremental step for compiler flag tuning, and the evaluation is good enough to take seriously, but not good enough to support the abstract's \"significantly outperforming\" claim. The GroupTuner comparison is not significant (p=0.174), and the GRPO group size K=8 was chosen on the evaluation programs, so the headline 1.732x is not a clean out-of-sample number.\n\nWhat's actually new: instead of a fixed, program-agnostic search bias (like PDCAT), WarmTuner learns a program-conditioned policy offline from historical tuning records, then refines the same policy online with GRPO using measured speedups. That is a sensible and useful combination. The paper ships a public artifact, reports three train/test splits, and does a set of ablations that mostly support the design. It also reports the GroupTuner p-value instead of hiding it, which is to its credit.\n\nSoft spots, in proportion:\n\n1. Overclaim in the abstract. The statistical test against GroupTuner is not significant. \"Significantly outperforming the compared techniques\" is false as written; it should say \"significantly outperforming RIO, SRTuner and PDCAT (not GroupTuner).\" This is a fixable wording issue.\n\n2. Test-set leakage in hyperparameter selection. The group-size study (Fig. 2) runs on the same testing programs as RQ1, and K=8 is then fixed for Table II. That makes the reported speedups test-set-informed. The paper discloses this in a footnote but does not treat it as a threat. This is the more serious issue: the exact magnitude of the advantage over GroupTuner could change if K were chosen differently. It does not invalidate the framework, but the quantitative claim should be labeled as optimistic.\n\n3. No error bars. Each cell is a median of five runs, but there are no confidence intervals or per-run spreads in Table II. For the main comparisons, especially the ones near significance, this matters.\n\n4. Minor: the offline data source is RIO records, which are weak by construction. The PDCAT-record variant helps, so this is less concerning, but the dependency on the data source's exploratory coverage is not deeply characterized.\n\nWho gets value: people working on compiler autotuning, and to a lesser extent anyone applying offline-to-online RL to program-level optimization. It doesn't reorganize a field, but it is a reasonable extension.\n\nRecommendation: send to peer review. Ask for the abstract to be reworded, the group-size selection to be moved to a validation set (or at least reported separately from the main results), and confidence intervals for the headline numbers.","headline":"A solid, incremental compiler-autotuning paper whose headline claim overreaches and whose group-size selection leaks test-set information; worth peer review but needs revision.","tokens_in":20526,"tokens_out":2842,"would_cite":true,"duration_ms":27121,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A tuner that starts from a program-specific policy learned offline and keeps adapting online reaches a 1.732x average speedup over GCC -O3.","keywords":["compiler autotuning","flag selection","offline-to-online reinforcement learning","GRPO","program-conditioned policy","GCC","warm start","CodeBERT"],"falsifier":"Train WarmTuner's offline policy on the same number of random flag configurations that carry no speedup signal (labels shuffled), then run online tuning on the test programs: if the average speedup at 1,250s is indistinguishable from the version pretrained on real historical records, then the offline warm start contributes nothing and the reported gains come from online search alone.","tokens_in":19708,"feed_emoji":"⚙️","tokens_out":4380,"duration_ms":38121,"temperature":0.7,"pith_summary":"The paper tries to establish that the starting point of compiler autotuning—not just the search—should be learned and adaptable. WarmTuner treats the prior as a trainable policy that predicts each flag's enable probability from source code, pretrains it on historical good configurations, and then updates the same policy online using measured speedups. The authors argue this solves the mismatch problem of existing tuners whose priors are fixed or program-agnostic. If correct, historical tuning data from any strategy can seed a per-program warm start, and online budgets are spent refining rather than rediscovering good regions. The evaluation reports 1.732x average speedup over -O3, best on 14/30 programs, with a significant edge over RIO, SRTuner, and PDCAT and a smaller edge over GroupTuner.","feed_headline":"Tuner that keeps learning online hits 1.73x speedup over GCC -O3","feed_subtitle":"A policy pretrained on past tuning data and refined per program outperforms four existing tuners under the same time budget.","key_machinery":"The central object is a program-conditioned factorized Bernoulli flag policy πθ(x|c_p, x_0). The program embedding c_p comes from averaging CodeBERT chunk embeddings of the source; x_0 is the -O3 flag state, encoded through per-flag state embeddings and an anchor embedding. The policy outputs per-flag enable probabilities qi = σ(zi), pretrained with binary cross-entropy against historical good configurations. Online, GRPO compares K=8 sampled candidates on the same program, computes group-normalized advantages A_i = (r_i - mean)/std, and updates the same policy with a clipped objective plus entropy regularization. This machinery unifies offline knowledge and online feedback in one trainable","core_discovery":"On the paper's own terms, WarmTuner demonstrates that compiler flag selection can be framed as an offline-to-online reinforcement learning problem. A factorized Bernoulli policy over 126 GCC flags, conditioned on a CodeBERT source embedding and the -O3 flag state, is first trained with per-flag binary cross-entropy on historical configurations that beat -O3. Then, on each target program, the same policy is refined with GRPO: each round samples eight configurations, measures their speedups, normalizes rewards within the group, and applies a clipped policy update. The paper claims this yields configurations faster than -O3 on all 30 test programs, with speedups from 1.133x to 4.159x and an ave","pith_inferences":["Editorial inference: the gain from offline pretraining probably depends on how well RIO's random search covered the flag combinations that matter for each program family; richer or more targeted historical data could widen or narrow the margin, and the paper only tests one alternative data source.","Editorial inference: averaging CodeBERT chunk embeddings may wash out cross-module structure; a representation that preserves call graphs or loop nests could strengthen the program conditioning further.","Editorial inference: because the improvement over GroupTuner is not statistically significant (p=0.174 at 5000s), the practical claim is best read as 'competitive and more consistent' rather than strictly dominant over the strongest baseline.","Editorial inference: a natural testable extension is to warm-start WarmTuner with records from GroupTuner or SRTuner rather than only RIO/PDCAT, and to measure whether structure-aware history yields even better initial distributions."],"forward_implications":["If correct, historical records from any tuner—RIO, PDCAT, or others—can be converted into a reusable program-specific policy, so past search effort is not wasted.","If correct, programs that differ sharply from the training distribution still get a tailored start, because the same policy is refined on the target program with real measured rewards.","If correct, the group-relative GRPO update is a good fit for compiler tuning because it needs no value model and uses cheap within-round comparisons.","If correct, the framework transfers beyond GCC flag selection to other optimization interfaces with a defined action space and measurable reward, such as pass ordering or inlining decisions.","If correct, the best measured configuration will typically include flags whose semantic role matches the target program's structure, as the jacobi-1d case suggests."],"fun_headline_variants":["RL tuner adapts per program, beats GCC -O3 by 1.73x","Offline-to-online RL tunes flags, 1.73x faster than -O3","WarmTuner: program-specific flag tuning hits 1.73x speedup","Tuner learns from history, refines live: 1.73x speedup"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole approach leans on the pretraining data—historical good configurations collected by random iterative optimization—containing enough signal about which flags matter for an unseen program; if that data is uninformative for a target family, the warm start is no better than random and the online budget may not be enough to catch up.","fun_headline_variants_meta":{"raw":{"variants":["RL tuner adapts per program, beats GCC -O3 by 1.73x","Offline-to-online RL tunes flags, 1.73x faster than -O3","WarmTuner: program-specific flag tuning hits 1.73x speedup","Tuner learns from history, refines live: 1.73x speedup"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000248,"raw_usage":{"total_tokens":1449,"prompt_tokens":876,"completion_tokens":573,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":487}},"tokens_in":620,"tokens_out":573,"duration_ms":5043,"temperature":1.0,"reasoning_tokens":487,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T01:20:26.883258+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train WarmTuner's offline policy on the same number of random flag configurations that carry no speedup signal (labels shuffled), then run online tuning on the test programs: if the average speedup at 1,250s is indistinguishable from the version pretrained on real historical records, then the offline warm start contributes nothing and the reported gains come from online search alone.","supporting_citations":[],"review_version":1}