{"id":"24697e55-c07f-4605-86d0-a553391b9495","arxiv_id":"2507.16649","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey of Profile Guided Optimization whose categorization is informative, but whose benchmark speedup tables contradict the accompanying prose.","lead":"This paper surveys Profile Guided Optimization, categorizes profiling methods and optimization stages, and adds a benchmark comparison of GCC and LLVM on x86 and ARM. The survey is a useful orientation, but the speedup data in the text repeatedly contradict the tables, so the empirical conclusions are not reliable as published.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The empirical core of the paper is self-contradictory: §5.3 prose disagrees with Tables 6/7 on key speedups and reports ARM AutoFDO results that §5.2 says were never collected; the headline performance/cost comparison is therefore unsupported as written.","rationale":"The reader's verdict is REJECT, and my reading supports it. The taxonomy sections on edge/path profiling, Ball-Larus instrumentation, MCF-based reconstruction, and GCC/LLVM pass integration are genuinely informative, but the paper's abstract and conclusion advertise an empirical comparison that is the basis for its central claim. That comparison cannot be trusted because the manuscript contradicts itself about its own measurements: prose values disagree with tables, and ARM AutoFDO values are discussed despite the setup excluding them. These are not matters of interpretation or disagreement with prior consensus; they are direct internal inconsistencies, so the claim fails on correctness grounds. I am not demanding perfect reproducibility of every benchmark cell; the decisive problem is that two versions of the data are presented as the same measurements, and the cost half of the headline claim has no supporting measurement at all. A rerun of the disputed rows, with raw logs published, would determine which version is correct and whether the conclusions can be restored. Until then, the published empirical claims are not reliable, and the reader's REJECT verdict stands.","tokens_in":31259,"tokens_out":6374,"duration_ms":62106,"concrete_test":"Reproduce the §5.2 protocol for the disputed SPECspeed 2017 rows (625.x264_s, 631.deepsjeng_s, 605.mcf_s, 600.perlbench_s, 657.xz_s) on the stated Intel Xeon Gold 6240C and Phytium FT-2000+/64 with GCC 12 and LLVM 15, and also record profile-collection/build overheads; publish the raw logs and the script that generates Tables 6/7. If the rerun matches the tables, the prose must be rewritten; if it matches the prose, the tables must be regenerated; if AutoFDO cannot be run on ARM under the stated setup, all ARM AutoFDO prose values must be removed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.3 cannot be reconciled with Tables 6 and 7. On AMD64, the prose reports 625.x264_s Clang-FDO at 1.087 and 631.deepsjeng_s GCC-FDO at 1.063, while Table 6 lists 1.000 and 0.985; the prose reports 605.mcf_s AutoFDO regressions of 0.982/0.976, while Table 6 shows gains of 1.034/1.086. On ARM, §5.2 explicitly limits the study to 'instrumentation-based PGO only on ARM architectures', yet §5.3 gives ARM AutoFDO values (0.971/0.959 for 605.mcf_s, 0.995 for 657.xz_s) that appear in neither table. The prose also claims FDO improves 'all' SPEC integer benchmarks, while Table 6 contains multiple entries below 1.0. Since the abstract and conclusion draw their headline comparison from these measurements, at least one of prose, tables, or experimental setup is wrong, and no artifact or raw data is provided to decide which. Separately, the claimed trade-off—instrumentation is costly, sampling is cheap—is not measured anywhere in §5: Tables 6/7 report only speedup ratios, not profile-collection time, build time, or runtime overhead. Thus the central performance/cost conclusion lacks a valid empirical basis.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper is a survey of Profile Guided Optimization (PGO/FDO) that organizes the field by profiling method (instrumentation vs. sampling), optimization stage (compile time, link/post-link time, runtime), compiler integration (GCC, LLVM), and target architecture. It reviews edge/path profiling, software and hardware sampling mechanisms (including IBS, PEBS, and ARM SPE), profile mapping and reconstruction, compiler passes, and post-link tools such as BOLT and Propeller. The paper then presents an empirical comparison of instrumentation-based FDO and sampling-based AutoFDO under GCC 12 and LLVM 15 on AMD64 and ARM, using SPEC CPU 2017 integer benchmarks, GROMACS, LAMMPS, FFTW, and microbenchmarks. The stated conclusions are that instrumentation-based FDO yields higher peak performance with more costly profile collection, while sampling-based AutoFDO gives more consistent speedups at lower deployment and runtime cost.","tokens_in":31501,"tokens_out":5452,"duration_ms":52912,"significance":"If the empirical core were reliable, the paper would be a useful one-stop reference: the taxonomy is broad, the algorithmic descriptions of Ball-Larus path profiling, minimum-cost-flow profile reconstruction, and the GCC/LLVM pass tables are informative, and the comparative evaluation would give practitioners concrete guidance. The survey portions alone have value, and the paper explicitly names modern systems (AutoFDO, BOLT, Propeller, CSSPGO, ARM SPE) with appropriate primary references. However, the significance of the contribution is dominated by the empirical claim, and that claim is internally inconsistent and lacks supporting overhead measurements. The strengths of the survey cannot compensate for an evaluation section whose prose and tables contradict each other on central numbers.","major_comments":[{"comment":"The prose in Section 5.3 cannot be reconciled with Table 6. The text reports 625.x264_s Clang-FDO at 1.087 and 631.deepsjeng_s GCC-FDO at 1.063, while Table 6 lists 1.000 and 0.985, respectively; it reports 605.mcf_s AutoFDO regressions of 0.982/0.976, while Table 6 shows GCC-AutoFDO 1.034 and Clang-AutoFDO 1.086. Since the abstract and conclusion draw their headline performance comparison from these measurements, at least one of the prose, the tables, or the experimental setup is wrong, and no artifact or raw data is provided to decide which.","section":"§5.3 vs. Table 6"},{"comment":"Section 5.2 explicitly states that the study evaluates \"instrumentation-based PGO only on ARM architectures,\" yet Section 5.3 reports ARM AutoFDO values (0.971/0.959 for 605.mcf_s and 0.995 for 657.xz_s) that appear in neither Table 7 nor the experimental setup. This is not a local wording issue: the comparative conclusions about sampling-based PGO on ARM depend on these undocumented numbers, and the contradiction indicates that the experiments may not have been run as described.","section":"§5.2 vs. §5.3"},{"comment":"The claim that \"instrumentation-based FDO consistently improves performance across all nine integer benchmarks\" and the later claim of \"robust, predictable speedups (2-10%) on all SPEC CPU 2017 integer benchmarks\" are directly contradicted by the paper's own tables. Table 6 shows GCC-FDO slowdowns for 623.xalancbmk_s (0.949), 631.deepsjeng_s (0.985), and 641.leela_s (0.940), and Clang-FDO slowdowns for 623.xalancbmk_s (0.938) and 657.xz_s (0.970). Table 7 shows GCC-FDO values below 1.0 for 623.xalancbmk_s (0.990), 631.deepsjeng_s (0.995), and 641.leela_s (0.961). These entries undermine the central empirical conclusion as stated.","section":"§5.3, Table 6, Table 7"},{"comment":"The paper's headline trade-off, that instrumentation-based PGO is costly while sampling-based PGO has much lower deployment and runtime costs, is not measured anywhere in Section 5. Tables 6 and 7 report only speedup ratios; there are no measurements of profile-collection phase duration, build time, or runtime overhead. Without such measurements, the conclusion in Section 7 about \"costly profile-collection phases\" and \"much lower deployment and runtime costs\" is unsupported by the presented evidence.","section":"§5, Tables 6-7, §7"},{"comment":"The experimental methodology is under-specified for the strength of the conclusions. Section 5.2 gives compiler versions, hardware, and the general FDO/AutoFDO workflows, but it does not state which SPEC inputs were used for profiling and for measurement, how many repetitions were performed, whether the reported ratios are geomeans or individual runs, or what variance was observed. Many reported values are within 1-3% of 1.0, so without this information the claims of consistent gains or regressions are not statistically grounded.","section":"§5.1-§5.2"}],"minor_comments":[{"comment":"The text says \"bars above the 1.0 line\" and \"these plots,\" but the results are presented in tables (Tables 6 and 7), not plots.","section":"§5.3"},{"comment":"The microbenchmark subsection lists seven kernels (acos, asin, bubble-sort, matrix inverse, L2-norm, matrix multiplication, SpMV) after saying \"six kernels,\" and the following sentence refers to \"These nine test cases,\" which does not match the count of the immediately preceding list.","section":"§5.1"},{"comment":"The benchmark name \"600.perlbech_s\" should be \"600.perlbench_s\".","section":"Table 6"},{"comment":"The citation \"nehao et al. [16]\" appears to refer to Dehao Chen et al.; the author names should be corrected throughout so that cited authors are identified by surname rather than given name.","section":"§3.2.3"},{"comment":"The closing lines \"Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009\" appear to be a template artifact and should be removed or replaced with the actual submission dates.","section":"Endmatter"},{"comment":"The reference list entry for [32, 32] contains a duplicated citation number; this should be cleaned up.","section":"Table 1"}],"recommendation":"reject","confidential_remarks":"The survey portions of the manuscript are potentially salvageable, but the empirical evaluation is the main claimed contribution and it is internally contradictory: the Section 5.3 prose disagrees with Tables 6 and 7 on multiple key speedups, reports ARM AutoFDO results that Section 5.2 says were never collected, and the paper provides no artifact or raw data. The central cost/performance trade-off is also not measured. These are load-bearing problems rather than presentation issues. If the authors were to rerun the experiments with a consistent methodology, report overhead measurements, and reconcile the prose with the tables, a substantially revised version could be reconsidered; in its current form I cannot recommend publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The survey portion of this paper is worth a read, but the experimental section is not trustworthy as written. Section 5.3's prose disagrees with Tables 6 and 7 on multiple speedup values, and it reports ARM AutoFDO numbers that the experimental setup says were never collected. The headline performance/cost conclusion rests on these numbers, so it falls apart.\n\nWhat is actually useful: the taxonomy of profiling techniques (edge/path instrumentation, software sampling, hardware sampling) is clean, and the comparison table for IBS, PEBS, and SPE is handy. The Ball-Larus path profiling explanation is correct and approachable. The compiler-pass tables for GCC 12 and LLVM 15 are a decent quick reference. The reference list spans 1971–2024 and a newcomer to PGO could genuinely orient themselves from Sections 2–4.\n\nThe soft spots are not minor. Section 5.3 says Clang-FDO gives x264_s 1.087 but Table 6 shows 1.000; it says GCC-FDO gives deepsjeng_s 1.063 but Table 6 shows 0.985; it claims AutoFDO regresses mcf_s at 0.982/0.976 while Table 6 shows gains of 1.034/1.086. On ARM, the prose reports AutoFDO values that Section 5.2 explicitly says were not collected and that appear in neither table. The abstract's claim that instrumentation-based FDO improves all nine integer benchmarks is contradicted by Table 6's sub-1.0 entries for xalancbmk (0.949) and leela (0.940). Separately, the conclusion's trade-off—instrumentation is costly, sampling is cheap—is never measured; Tables 6 and 7 report only speedup ratios, not collection time or overhead. This is a load-bearing empirical section with no reliable basis.\n\nWho is this for? Compiler engineers and graduate students wanting a survey of profiling techniques and compiler passes. They should read Sections 2–4 and ignore Section 5 until it is fixed. The survey scope is useful and the problems are correctable, so the paper deserves a serious referee—but the current submission should not be accepted without a full rewrite of the experiments and a reconciliation of prose, tables, and setup. If I were the editor, I'd send it to review with a clear expectation of major revision and a request for the benchmark scripts and raw measurements.","headline":"The survey half is a genuinely useful reference, but Section 5 contradicts its own tables so badly that the paper's central empirical claims don't survive.","tokens_in":32061,"tokens_out":2596,"would_cite":false,"duration_ms":26689,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey of five decades of profile-guided optimization claims the field reduces to a single trade-off — instrumentation buys higher peak speed, sampling buys consistent gains at far lower cost — and benchmarks both across x86-64 and…","keywords":["profile guided optimization","feedback-directed optimization","instrumentation","hardware sampling","AutoFDO","compiler optimization","LLVM","GCC"],"falsifier":"Re-run one reported configuration exactly as described — for example, build 625.x264_s with Clang 15 instrumented FDO on an Intel Xeon Gold 6240C and measure its speedup over a plain -O3 build. Table 6 reports 1.000 for this case while Section 5.3's prose reports 1.087; a measurement matching neither value would show the reported data are not reproducible, and a full re-run of the benchmark matrix would settle which of the two conflicting accounts is correct.","tokens_in":31040,"feed_emoji":"⚙️","tokens_out":14140,"duration_ms":127007,"temperature":0.7,"pith_summary":"This paper tries to establish a complete map of Profile Guided Optimization (PGO) — the family of techniques that runs a program once to learn how it behaves, then recompiles it with those observations — and to settle, with its own measurements, which flavour a practitioner should choose. On the survey side, it organizes five decades of work into a taxonomy built on how profiles are collected (exact instrumentation versus statistical sampling, hardware- or software-assisted) and when the feedback is applied (compile time, link or post-link time, runtime). On the evidence side, it reports speedups for four build modes — GCC and Clang, each with instrumentation-based FDO and sampling-based AutoFDO — across SPEC CPU 2017 and other workloads on x86-64 and ARM64. Its conclusion is that instrumentation-based PGO reaches higher peak performance but demands an expensive train-then-recompile cycle, while sampling-based PGO delivers consistent gains at much lower deployment cost. If the survey is right, it works as a one-stop decision guide for compiler and performance engineers, and its open challenges — zero-overhead sampling, dynamic workloads, and cross-architecture portability — become the field's agenda.","feed_headline":"Instrumentation wins peak speed; sampling wins on cost","feed_subtitle":"A five-decade PGO survey benchmarks both modes head-to-head on x86 and ARM to pick the right one.","key_machinery":"The load-bearing machinery is the profile-reconstruction pipeline that turns raw observations into compiler decisions. On the instrumentation side, the Ball-Larus algorithms are central: optimal edge-counter placement is reduced to a maximum-spanning-tree problem, with flow-conservation equations at each vertex reconstructing all edge frequencies from a minimal counter set, and path profiling assigns each acyclic path a unique integer index by instrumenting only the chords of a spanning tree, giving complete path counts at roughly 1.3 times the cost of edge profiling. On the sampling side, the anchor is minimum-cost-flow (MCF) inference: hardware samples give sparse, noisy counts for a subset of edges, and a primal-dual successive-shortest-path solver (the paper's Algorithm 2) fills in a full, control-flow-graph-consistent frequency assignment that minimizes a cost whose edge weights encode confidence in the samples — the step credited with letting sampling-based PGO approach instrumentation accuracy. Around these two cores sit the integration objects: GCC's .gcno/.gcda and LLVM's .profraw/.profdata profile formats, the compiler passes that consume reconstructed weights (branch reordering, block placement, inlining, indirect-call promotion, loop transformations), and the post-link tools BOLT and Propeller that reorder the final binary.","core_discovery":"The paper's central claim is that the whole PGO landscape reduces to one engineering trade-off with a well-studied middle ground. Instrumentation inserts counters into the compiled program, yielding exact edge and path counts at the price of slowing the training run; sampling, especially hardware sampling through Intel PEBS, AMD IBS, branch-recording (LBR), and ARM SPE, gathers statistical evidence at a few percent overhead, and minimum-cost-flow inference reconstructs full control-flow-consistent frequencies from those sparse samples. The paper argues that this reconstruction machinery closes most of the quality gap, so sampling-based AutoFDO becomes competitive with instrumented FDO on compute-bound code while staying far cheaper to deploy. Its SPEC CPU 2017 measurements are offered as evidence that instrumentation-based FDO meets or beats sampling on peak speedup on both x86-64 and ARM64, with neither GCC nor LLVM systematically ahead. The survey also claims that ARM64 is a viable PGO target because SPE provides precise, low-overhead, zero-skid sampling, and that the next bottlenecks are cutting sampling overhead, keeping profiles valid when training inputs drift from production workloads, and making PGO portable across architectures.","pith_inferences":["Beyond the paper: the evidence it reports points toward a hybrid it never tests — instrumented profiles for the few hot kernels that justify training cost, sampled profiles for everything else — which its 'duplex sampling' future-work sketch gestures at.","The survey's cross-architecture sampling conclusion is stronger than its own ARM data: it reports instrumentation-based results on ARM64 but no AutoFDO measurements there, so the claim that SPE makes ARM sampling competitive rests on cited prior work, not on the paper's experiments.","A testable reading of the reconstruction literature it surveys: profile quality, not collection method, predicts PGO gains; one could verify the paper's central trade-off by degrading an instrumented profile to sampling-level noise and checking whether speedups converge.","Its fixed-input benchmark methodology leaves input robustness open; a natural follow-up would measure how speedups decay as test inputs diverge from the training inputs that produced the profiles."],"forward_implications":["A practitioner choosing among GCC or Clang and FDO or AutoFDO gets a concrete answer: use instrumented FDO when a training run is affordable and peak speed matters, and sampling-based AutoFDO when deployment cost dominates.","Sampling is not a free lunch: on pointer-chasing, memory-bound code such as 605.mcf_s and sparse matrix-vector multiplication, AutoFDO can regress performance instead of improving it.","ARM64 becomes a first-class PGO target, since SPE-style sampling is claimed to support production AutoFDO workflows on ARM the way PEBS and IBS do on x86.","The proposed taxonomy — profiling method times optimization stage — is offered as the organizing frame for both future PGO research and tooling decisions.","Because neither compiler systematically dominates, workload characteristics, not toolchain choice, are what decide how much PGO helps."],"supporting_citations":[{"why":"Ball and Larus's efficient path profiling; supplies the incremental-encoding algorithm at the core of the instrumentation branch.","marker":"[7]"},{"why":"Ball and Larus's optimal counter placement; reduces edge profiling to a spanning-tree problem so all frequencies follow from minimal counters.","marker":"[8]"},{"why":"Chen et al.'s AutoFDO; defines the sampling-based PGO workflow that the paper benchmarks against instrumentation.","marker":"[15]"},{"why":"He et al.'s profile inference; the theoretical foundation the paper credits for reconstructing CFG-consistent profiles from sparse samples.","marker":"[24]"},{"why":"Levin et al.'s minimum-cost circulation method; the direct ancestor of the paper's Algorithm 2 for repairing missing and inaccurate samples.","marker":"[33]"},{"why":"Dean et al.'s ProfileMe; the instruction-level sampling co-design that PEBS and IBS hardware sampling build upon.","marker":"[19]"},{"why":"Miksits et al.'s ARM SPE study; the cited evidence that SPE-driven PGO reaches over 99 percent hotspot fidelity at under 5 percent overhead.","marker":"[39]"},{"why":"Panchenko et al.'s BOLT; the post-link sampling-driven binary optimizer used to demonstrate binary-level PGO.","marker":"[42]"},{"why":"Shen et al.'s Propeller; the link-time relinking optimizer whose basic-block sections enable profile-driven ordering at link time.","marker":"[46]"},{"why":"Ayupov et al.'s stale profile matching; the 2024 work cited for keeping profiles valid as binaries evolve.","marker":"[6]"}],"fun_headline_variants":["Sampling-based PGO nearly matches instrumentation","Instrumentation peaks; sampling saves cost","Five-decade PGO survey: instrumentation vs sampling","Sampling closes most of PGO performance gap","Sampling PGO: low overhead, near-instrumentation speed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's empirical conclusions rest on the speedup numbers in Tables 6 and 7 being accurate and internally consistent, and the surrounding prose reports several values that do not match those tables, so the architecture-level comparison of instrumentation versus sampling stands or falls with the reliability of those measurements.","fun_headline_variants_meta":{"raw":{"variants":["Sampling-based PGO nearly matches instrumentation","Instrumentation peaks; sampling saves cost","Five-decade PGO survey: instrumentation vs sampling","Sampling closes most of PGO performance gap","Sampling PGO: low overhead, near-instrumentation speed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000978,"raw_usage":{"total_tokens":4138,"prompt_tokens":917,"completion_tokens":3221,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":3148}},"tokens_in":533,"tokens_out":3221,"duration_ms":23187,"temperature":1.0,"reasoning_tokens":3148,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:04:17.001184+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run one reported configuration exactly as described — for example, build 625.x264_s with Clang 15 instrumented FDO on an Intel Xeon Gold 6240C and measure its speedup over a plain -O3 build. Table 6 reports 1.000 for this case while Section 5.3's prose reports 1.087; a measurement matching neither value would show the reported data are not reproducible, and a full re-run of the benchmark matrix would settle which of the two conflicting accounts is correct.","supporting_citations":[{"cited_title":"Ball and J.R","cited_arxiv_id":null,"evidence_quote":"Ball and Larus's efficient path profiling; supplies the incremental-encoding algorithm at the core of the instrumentation branch."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Ball and Larus's optimal counter placement; reduces edge profiling to a spanning-tree problem so all frequencies follow from minimal counters."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Chen et al.'s AutoFDO; defines the sampling-based PGO workflow that the paper benchmarks against instrumentation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"He et al.'s profile inference; the theoretical foundation the paper credits for reconstructing CFG-consistent profiles from sparse samples."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Levin et al.'s minimum-cost circulation method; the direct ancestor of the paper's Algorithm 2 for repairing missing and inaccurate samples."},{"cited_title":"Dean, J.E","cited_arxiv_id":null,"evidence_quote":"Dean et al.'s ProfileMe; the instruction-level sampling co-design that PEBS and IBS hardware sampling build upon."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Panchenko et al.'s BOLT; the post-link sampling-driven binary optimizer used to demonstrate binary-level PGO."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Ayupov et al.'s stale profile matching; the 2024 work cited for keeping profiles valid as binaries evolve."}],"review_version":1}