{"id":"6ae87dda-ec71-44bd-9aef-1a163ec31b71","arxiv_id":"2505.08598","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"GroupTuner, which performs simulated-annealing search by mutating coherent groups of GCC optimization flags derived from pass structure, reports 12.39% average improvement over -O3 across 36 benchmarks, beating BOCA, CFSCA, SRTuner, RIO, and SA.","lead":"This paper presents GroupTuner, a compiler auto-tuning method that mutates groups of related optimization flags, derived from the compiler's internal pass structure, instead of trying to identify the most important flags. In tests on 36 C programs with GCC, it reported an average 12.39% performance gain over -O3 and used less tuning time than five comparison methods.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GroupTuner's headline gains over BOCA may come from search budget and seeded initialization rather than from the pass-derived grouping; the paper's own comparison baseline differs across methods and no grouping ablation is reported.","rationale":"The reader's weakest assumption identifies exactly the same gap: the functional-coherence premise of the pass-derived grouping is asserted (Section 4.2) but not validated against alternative groupings. I agree there is no fatal internal inconsistency; the architecture, the Algorithm 1 pseudo-code, and the reported experiments are coherent, and RQ4 shows some groups contribute more than others. The concern is empirical identifyability: the reported gains are averaged over a single run per benchmark (no seeds, no error bars), and the comparison against BOCA/CFSCA mixes algorithm differences with grouping differences and with search-budget differences. The paper's own Table 4 extension to 1,000 rounds and Table 5 cross-version results show GroupTuner still leads BOCA, but those runs still lack grouping ablations and still conflate the SA schedule with the group prior. Thus the central mechanism claim of 'group-aware' is not established by the data as reported. A grouped-mutation ablation is the single decisive experiment. If that ablation succeeds, the paper's conditional acceptance becomes an accept; if it fails, the method reduces to a well-seeded simulated annealer with a history-guided candidate list, and the novelty claim weakens substantially. I agree with the reader's CONDITIONAL verdict and with the request for the same ablation.","tokens_in":20283,"tokens_out":1675,"duration_ms":15648,"concrete_test":"Run a controlled ablation on the 36 benchmarks with identical SA hyperparameters (T0, cool_r, alpha, round_init) and identical candidate-list size, varying only the mutation partition: (a) the 15 pass-derived groups; (b) 15 random groups of matched sizes, repeated over 5 random seeds; (c) 6 and 30 random groups; and (d) a variant where the selected 'group' is a single randomly chosen option (group size 1). If (a) does not significantly beat (b)-(d) in mean improvement over -O3 with matched 500-iteration budgets and matched wall-clock overhead, then the pass-derived grouping is not the load-bearing component of the reported 1.83% edge over BOCA.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that group-aware mutation based on GCC pass-derived option groups is the cause of GroupTuner's 12.39% average improvement and 1.83% edge over BOCA. The most load-bearing unvalidated assumption is that the specific 15-group clustering, derived from pass structure, is what drives the advantage. The paper reports no ablation against alternative groupings: random groups of matched size, larger/smaller group counts, or data-driven clusters. Sections 4.2 and 5.6 argue for functional coherence but only demonstrate that some groups (LOOP 1, GIMPLE 2) contribute more, not that the pass-derived grouping outperforms other partitions. RQ3 is also confounded: BOCA and CFSCA re-train models each iteration, which explains their higher wall-clock cost, but the paper does not equalize iteration counts or model-training time when claiming efficiency. Additionally, the SA comparison in Section 5.2 is described as 'configured similarly' but mutates globally; since T0, cool_r, alpha, and round_init are not reported anywhere, it is possible GroupTuner's advantage is partly from a better-tuned annealing schedule rather than from grouping. Section 6.1 concedes the grouping is GCC-specific and cannot be directly transferred to LLVM, so the practical scope of the claimed advantage is narrow. The paper's own limitations and missing ablation mean the central mechanism, as opposed to the overall tuner, is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GroupTuner, a GCC compiler-option auto-tuner that statically partitions 206 optimization options into 15 groups derived from GCC pass structure, then performs a simulated-annealing-style search that mutates one group at a time, seeded from the -O3 configuration. Across 36 benchmarks from cBench, PolyBench, and SPEC CPU2017 with 500 tuning iterations, GroupTuner reports the best improvement over -O3 on 27/36 programs, an average improvement of 12.39%, compared with 10.56% for BOCA, while using 77.21% of RIO's wall-clock time. Additional experiments report 1000-iteration behavior and results on GCC 11.5.0 and 13.1.0.","tokens_in":20517,"tokens_out":6595,"duration_ms":66190,"significance":"If the mechanism claim holds, GroupTuner is a useful contribution: it replaces expensive per-iteration model training with a cheap, compiler-structure-derived search prior, and its cross-version and long-run experiments suggest robustness beyond a single compiler snapshot. The paper has notable strengths: an available artifact, careful measurement noise control (Turbo Boost off, core pinning, five-run averages, output-correctness checks), comparisons against five methods using their own code, and a clean separation of the grouping construction from benchmark outcomes, so the approach is not circularly fitted to the evaluation set. The main weakness is that the causal role of the pass-derived grouping is not isolated, and the headline numerical margins lack repeated-run statistics.","major_comments":[{"comment":"The central claim is that functionally coherent, pass-derived groups are what make GroupTuner successful. The evaluation never compares the 15 pass-derived groups against alternative partitions, such as random partitions of the same 206 options into 15 groups of matched size, different group counts, or data-driven clusters. The RQ4 analysis in Figure 9 only shows that mutating some groups is associated with larger improvements during the actual search; it does not show that the pass-derived grouping outperforms other partitions. Without such a control, the observed advantage over global SA could be due to any constrained local-mutation scheme.","section":"§4.2, §5.6"},{"comment":"Algorithm 1 depends on T0, Tmin, coolr, alpha, round_init, and the candidate-list size, and Section 5.2 states that the SA baseline is 'configured similarly' but mutates globally. None of these values is reported anywhere in the paper. If the two methods use different annealing schedules or different initialization rounds, the comparison is confounded. Please report the exact hyperparameters and include a sensitivity analysis, or compare GroupTuner with a global-mutation variant that uses the identical schedule.","section":"§5.2, Algorithm 1"},{"comment":"Table 3 reports a single best-observed percentage per program for each algorithm. All compared methods are stochastic in candidate selection and mutation, so a single run per program cannot establish the 1.83-percentage-point average margin over BOCA, nor can it support the claim that GroupTuner is best on 75% of programs. I request multiple seeds with confidence intervals, or at least a paired statistical test over programs for the average improvement, for the main 500-iteration results and for Tables 4 and 5.","section":"Table 3"},{"comment":"The comparison is partly confounded by initialization: Section 5.4 notes that GroupTuner starts from -O3, while BOCA, SRTuner, and CFSCA begin from randomly generated combinations. The early and possibly final advantages of GroupTuner are therefore inherited in part from the expert-designed -O3 starting point rather than from the group-aware mutation mechanism. The paper should add a randomly initialized GroupTuner variant, or re-run all baselines from the same -O3 seed, to isolate the contribution of the grouping and history-guided search.","section":"§5.4, Figure 5"}],"minor_comments":[{"comment":"Section 5.1 says the number of iterations is set to 500, while Section 5.2 says the three state-of-the-art methods are run with 'the source code and the default settings they provide.' Please clarify whether the default iteration counts of BOCA, CFSCA, and SRTuner were overridden to 500.","section":"§5.1, §5.2"},{"comment":"Table 4 reports 1000-iteration values but does not give the corresponding 500-iteration values for the same subset of programs, so the claim that GroupTuner 'continues to discover improved optimization combinations beyond 500 iterations' cannot be directly verified from the table; please include both columns.","section":"§6.2, Table 4"},{"comment":"The metric in Figure 9, 'average performance contribution' of each group, is underspecified: it should be stated exactly how per-iteration improvements are attributed to a group, given that the base combination varies across iterations.","section":"§5.6, Figure 9"},{"comment":"There are duplicated and mis-referenced labels in the text: 'Listing 1' appears twice for different listings, and Figure 10 is referenced before the listing it refers to. Please renumber the listings and figures consistently.","section":"§2, Listing 1"},{"comment":"The terms 'round' and 'iteration' are used interchangeably; please unify them, and define in Algorithm 1 that 'perf' denotes execution time so that lower values are better.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical systems paper that fits LCTES well. The cross-version and long-run experiments are strengths, and the missing grouping ablation is the key scientific gap; I see it as fixable rather than fatal. I would ask the editor to insist on the hyperparameter disclosure and the grouping/initialization controls before acceptance, since the paper's central mechanism claim currently rests on confounded comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"GroupTuner is a decent empirical paper, but the mechanism claim outruns the evidence. The new piece is grouping GCC's flags by pass structure (via CodeQL) and mutating one group at a time, seeded from -O3 and historically good candidates. It is a sensible block-coordinate heuristic, and the experiments are more careful than most compiler auto-tuning papers: Turbo Boost off, core pinning, five-run averages, correctness checks, comparisons against five methods using the authors' own code. The 12.39% average gain over -O3 and the 39.25% time ratio versus BOCA are believable measurements.\n\nThe soft spots are about isolating cause. There is no ablation against random groups, different group counts, or data-driven clusters, so we do not know whether pass-structure grouping is what drives the advantage. The SA baseline is 'configured similarly' but the annealing parameters (T0, Tmin, coolr, alpha, round_init) are never reported, so part of the edge could come from the schedule rather than the grouping. The -O3 seed is also a plausible contributor; the paper does not vary it. Table 3 has one best-observed number per program with no error bars or significance tests, and some margins are thin. RQ4's group-contribution analysis is post hoc and does not validate the grouping. The GCC-specific grouping limits transfer, a point the paper honestly concedes.\n\nOne correction to the stress-test note: iteration counts are equalized at 500 across methods, so the efficiency comparison is not confounded by different search budgets. The wall-clock overhead of model retraining is a real cost, not an artifact.\n\nOn the credit side, the artifact is available, the cross-version (GCC 11/13) and 1000-iteration checks show the result is not a fluke of one compiler or one budget, and the grouping is derived from compiler source, not from benchmark outcomes, so circularity is low.\n\nThis paper is for compiler auto-tuning researchers and for practitioners who want low-overhead flag specialization. It deserves peer review; my recommendation is major revision with a grouping ablation, a sensitivity analysis of group size, significance tests or at least per-program variances, and full parameter disclosure. Without that, the central mechanism stays conditional.","headline":"GroupTuner is a solid empirical compiler tuning paper whose group-aware mechanism is plausible but not yet isolated from seeding and annealing choices; it deserves a referee who asks for an ablation.","tokens_in":21163,"tokens_out":3631,"would_cite":true,"duration_ms":34566,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Grouping compiler flags by the optimization passes that use them, then mutating one group at a time, finds better GCC option combinations in less time than methods that first try to infer which individual options matter.","keywords":["compiler auto-tuning","GCC optimization options","option grouping","pass-based grouping","group mutation","simulated annealing","iterative optimization","performance tuning"],"falsifier":"Run GroupTuner on the same 36 benchmarks with 15 equally sized groups formed by random permutation of the 206 flags instead of pass-coherence grouping, keeping every other setting fixed; if the random-group version matches or beats the pass-group version, the proposed grouping mechanism is not the source of the reported advantage. Also check a program from the paper's case study (telecom_gsm) where -ftree-loop-if-convert hurts when combined with -ftree-loop-vectorize; GroupTuner should still find the configuration that enables only the vectorizer, since both flags sit in the same Loop-1 group.","tokens_in":1696,"feed_emoji":"⚙️","tokens_out":2343,"duration_ms":104150,"temperature":0.7,"pith_summary":"The paper claims that when tuning GCC's optimization options for a particular program, it is more effective to mutate coherent groups of flags than to first identify which individual flags are critical. It proposes GroupTuner, which statically groups 206 GCC options into 15 clusters according to the compiler passes that gate or execute them, and then repeatedly mutates one randomly chosen group within the best option combinations found so far, guided by a simulated-annealing acceptance rule. Across 36 benchmark programs, the paper reports the best-performing combination in 75% of cases and an average gain of 12.39% over -O3, surpassing the strongest existing method by 1.83 percentage points while taking roughly 2.5x less tuning time. If this holds, compiler auto-tuning can be built on pass-structure priors rather than fragile importance inference in a sparse, noisy, high-dimensional search space.","feed_headline":"Grouped flag search beats the best tuner by 1.83% in less time","feed_subtitle":"Avoids picking 'critical' flags; averages 12.39% speedup over -O3 across 36 programs.","key_machinery":"The load-bearing mechanism is the pair of a static option grouping and a history-guided group-mutation search. The grouping is built once per compiler version by static analysis of GCC's source: locate each option's use sites, associate options with the pass that gates or executes them, merge groups sharing options, and assign leftovers to the nearest group according to the pass order in gcc/passes.def, producing 15 groups of 4-28 options. The search initializes from -O3 and the candidate list of the ten best configurations; each iteration selects a candidate at random, picks a group at random, flips each option in that group with probability 1/2, and keeps worse results with probability $e^{-\\Delta/(T\\alpha)}$ under a cooling temperature. This combination keeps adjustments locally coherent: the unchanged groups preserve the base configuration's validated behavior, while the mutated group can be explored for internal synergies without perturbing the rest of the vector.","core_discovery":"GroupTuner's central claim is that explicit critical-option identification is not a necessary step in compiler auto-tuning. The method maps every GCC optimization flag to the optimization pass or passes that read it (using static analysis of GCC internals), merges overlapping option-pass clusters, and assigns ungrouped flags to nearby groups by pipeline order, yielding 15 functionally coherent groups. The search then keeps a list of the ten best full option configurations observed, randomly picks one as a base, mutates only the flags inside a single randomly chosen group, and retains the result either because it improves the list or probabilistically through simulated annealing. The paper argues that this preserves the proven-good parts of earlier combinations while concentrating exploration where functional relatedness makes synergistic interactions likely, and reports that the resulting combinations outperform -O3 by an average of 12.39%, beat the strongest baseline (Bayesian optimization with random-forest importance inference) by 1.83 percentage points, and finish in only 39.25% of that baseline's average tuning time.","pith_inferences":["A direct way to test the paper's explanation is to rerun the same search with randomly permuted groups of identical sizes; if random groups perform just as well, group coherence is not what carries the gains, even if GroupTuner still wins.","The pass-based grouping is GCC-specific, but the same principle could be realized in pass-reorderable compilers by learning groups from program IR transformations or observed flag interactions across a corpus, a step the paper leaves to future work.","The efficiency gap is likely to widen as program runtimes grow, since GroupTuner's per-iteration overhead is near zero while model-based methods retrain each round, so the measured 2.5x advantage may understate the practical benefit on long-running workloads."],"forward_implications":["Programs can be auto-tuned with more headroom: the reported 12.39% average gain over -O3 comes on top of a default that GCC itself chose, so the practical benefit is real speedup without manual flag expertise.","Tuning budgets become smaller: at 77.21% of random-search time and 39.25% of the strongest model-based baseline's time, per-program tuning becomes viable for programs with longer running times.","A single static grouping serves all programs on a given compiler version, so new programs do not trigger additional grouping cost during tuning.","Loop and GIMPLE-phase option groups carry the largest share of observed gains (averaging 5.05% and 2.83%), pointing users to where optimization effort pays off for GCC-like pipelines."],"supporting_citations":[{"why":"Supplies the strongest baseline—Bayesian optimization with random-forest critical-option inference—that GroupTuner must outperform.","marker":"[8]"},{"why":"Provides a synergistic-relations baseline and the predecessor implementation that the authors extend.","marker":"[23]"},{"why":"Supplies the static-analysis machinery used to map each option to the pass that gates or executes it.","marker":"[14]"},{"why":"Supplies the simulated-annealing acceptance rule used to decide whether to keep worse candidate combinations.","marker":"[6]"},{"why":"Supplies the block-coordinate-descent motivation for partitioning a high-dimensional variable space into blocks.","marker":"[27]"},{"why":"Establishes that fixed default optimization levels do not generalize across programs, motivating per-program tuning.","marker":"[20]"},{"why":"Supplies the random-iterative-optimization baseline against which tuning time is normalized.","marker":"[16]"}],"fun_headline_variants":["Grouped flag mutations outsmart compiler tuners by 1.83%","GroupTuner beats state-of-the-art without picking key flags","Compiler tuning finds 12.39% gain by mutating flag groups","Group-aware search skips critical flags, wins by 1.83%","Flag group mutation outperforms Bayesian tuning by 1.83%"],"cache_read_input_tokens":23168,"weakest_assumption_plain":"The premise that flags used by the same pass or the same optimization stage are functionally related enough that mutating one such group at a time preserves gains and discovers synergies; the paper does not compare this pass-based grouping against random or data-driven groupings of the same size.","fun_headline_variants_meta":{"raw":{"variants":["Grouped flag mutations outsmart compiler tuners by 1.83%","GroupTuner beats state-of-the-art without picking key flags","Compiler tuning finds 12.39% gain by mutating flag groups","Group-aware search skips critical flags, wins by 1.83%","Flag group mutation outperforms Bayesian tuning by 1.83%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000493,"raw_usage":{"total_tokens":2449,"prompt_tokens":997,"completion_tokens":1452,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":1357}},"tokens_in":613,"tokens_out":1452,"duration_ms":9864,"temperature":1.0,"reasoning_tokens":1357,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:51:18.541874+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run GroupTuner on the same 36 benchmarks with 15 equally sized groups formed by random permutation of the 206 flags instead of pass-coherence grouping, keeping every other setting fixed; if the random-group version matches or beats the pass-group version, the proposed grouping mechanism is not the source of the reported advantage. Also check a program from the paper's case study (telecom_gsm) where -ftree-loop-if-convert hurts when combined with -ftree-loop-vectorize; GroupTuner should still find the configuration that enables only the vectorizer, since both flags sit in the same Loop-1 group.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the strongest baseline—Bayesian optimization with random-forest critical-option inference—that GroupTuner must outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides a synergistic-relations baseline and the predecessor implementation that the authors extend."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the static-analysis machinery used to map each option to the pass that gates or executes it."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the simulated-annealing acceptance rule used to decide whether to keep worse candidate combinations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the block-coordinate-descent motivation for partitioning a high-dimensional variable space into blocks."},{"cited_title":"Hoste and L","cited_arxiv_id":null,"evidence_quote":"Establishes that fixed default optimization levels do not generalize across programs, motivating per-program tuning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the random-iterative-optimization baseline against which tuning time is normalized."}],"review_version":1}