{"id":"4b0feccf-fb2e-415d-9fd5-ec33caa4676a","arxiv_id":"2608.05855","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RepoOMP combines dependency-graph evidence, rule-based routing, and bounded LLM context to parallelize OpenMP hotspots in large repositories with better acceptance, speedup, and token efficiency than unstructured agent baselines.","lead":"RepoOMP is a tool that finds hot loops in large code repositories and adds OpenMP parallelization directives, first mapping file and data dependencies and then sending each candidate either to deterministic rules or to an AI agent with a compact context. In tests on 951 hotspots it reports 372 accepted parallel versions, 330 from real-world codebases, with a median 2.25x speedup and lower AI token cost than an unstructured baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-hotspot validation oracle for the 330 accepted real-world hotspots is unspecified, and the funnel counts are internally inconsistent (817 vs 789 attempted), so the headline acceptance statistics are not yet reproducible.","rationale":"The reader's CONDITIONAL verdict is appropriate, and my concern does not overturn it; it sharpens it. The paper is an honest systems evaluation with explicit denominators, distributions, ablations, and limitations, and the MAP/router/STC design is coherent and worth building on. However, the most load-bearing part of the central claim is the accepted-set statistics: 372 accepted hotspots, 330 real-world, median 2.25x, and the acceptance-rate percentages in Table 12. Those statistics inherit their meaning from the per-hotspot validation oracle, and the paper only specifies oracles for the nine detailed workload commands plus the suites. The artifact is not inspectable in the preprint, so one cannot yet verify that each of the other 321 accepted real-world hotspots actually passed a workload-specific correctness check rather than merely compiling and showing speedup. The funnel inconsistency (817 vs 789 attempted; 212 vs 184 rollbacks) is a concrete symptom that the reported counts have not been reconciled against the row-level tables. A per-candidate log audit would settle both issues. This is not a demand for formal proof; it is a request for the evidence trail that the paper itself says it emits (Section 3.5 lists serialized MAP, router labels, STC prompt records, patches, validation logs, and timing logs). If the audit shows every accepted hotspot has a real, exercising executable check and the funnel reconciles, the central claim stands as stated. If not, the accepted-set statistics need revision.","tokens_in":26470,"tokens_out":11118,"duration_ms":107346,"concrete_test":"From the released artifact, extract the per-candidate validation record for every one of the 330 accepted real-world hotspots: workload command, input asset, golden-reference oracle, pass/fail log, and timing log. Then select one non-detailed accepted hotspot per repository, revert exactly that patch, and rerun the corresponding workload command; if the program output is unchanged, the check did not exercise the hotspot and the acceptance predicate is vacuous for that candidate. Independently, recompute the funnel from per-candidate logs: sum the attempted, compile-failure, check-failure, rollback, and accepted counts per workload and reconcile them with the abstract's 951/372 and Section 4.1's 817/445 numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—330 accepted real-world hotspots with median 2.25x speedup—rests on an acceptance predicate that the paper never fully specifies at per-hotspot granularity. Section 3.5 and Table 3 define validation oracles for the nine workload commands (FFmpeg-MPEG4/NLMeans/Deshake, NCNN-MobNet/ShuffNet/ResNet, GROMACS-Nonbd/PME/LINCS) and for NPB/BOTS, but no table or statement maps each of the 330 accepted real-world hotspots to a specific executable check, input asset, golden baseline, or oracle. If a hotspot is accepted under a workload command whose output path does not exercise it, the 'workload-specific check' is vacuous and acceptance reduces to compilation plus speedup. The paper's Section 5 disclaimer that acceptance is not proof of race freedom does not repair a missing per-hotspot correctness check. There is also a direct internal inconsistency in the funnel: Section 4.1 says 817 hotspots enter attempted transformation, while Table 14 rows sum to 789 attempted (67+10+461+114+137); RQ2 reports 212 rollbacks, while Table 14 sums to 184. This changes the headline real-world acceptance rate from 372/817=45.5% to 372/789=47.2% and means the reported counts need reconciliation before the accepted-set statistics can be trusted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes RepoOMP, a hybrid workflow for OpenMP parallelization of hot loops in large code repositories. It constructs a Multi-granularity Attributes Performance graph (MAP) from a compilation database and syntactic analysis, routes each hotspot to a deterministic rule engine or an LLM agent through a Rule-Agent Router, and compresses the evidence into a Structured Transformation Context (STC). Generated transformations are accepted only if they compile, pass workload-specific executable checks, and show positive speedup. The evaluation reports 372 accepted hotspots out of 951 profiled, including 330 real-world hotspots from FFmpeg, NCNN, and GROMACS, with a median accepted-set speedup of 2.25x; average speedups of 8.23x on NPB and 8.96x on BOTS; and, for nine selected real-world kernels, a cross-backbone mean speedup of 5.25x and reduced agent-side token cost relative to an unstructured Claude Code baseline.","tokens_in":26675,"tokens_out":12146,"duration_ms":99808,"significance":"If the quantitative claims survive repair, this is a valuable systems contribution: the paper identifies a real evidence-recovery gap, proposes an inspectable three-stage architecture, and evaluates it over a large 951-hotspot funnel with explicit disclosure of accepted-set distributions, repeated-run spreads, ablation results, and limitations. The matched-backbone protocol, token accounting, and the authors' willingness to state what the evaluation does not establish are notable strengths. However, the headline acceptance statistics currently rest on internally inconsistent funnel counts and on an unspecified per-hotspot validation mapping for the 330 accepted real-world hotspots; until those are supplied, the central quantitative claims are not reproducible from the manuscript.","major_comments":[{"comment":"The hotspot funnel is internally inconsistent. Section 4.1 states that 817 hotspots enter attempted transformation, but the Attempted column of Table 14 sums to 789 (67+10+461+114+137). Section 4.3 reports 212 no-speedup rollbacks, while the RB column of Table 14 sums to 187. For the real-world rows, Section 4.1 reports 110 workload-check failures and 157 no-speedup rollbacks, but Table 14 sums to 100 check failures and 170 rollbacks for those rows. Table 14's GROMACS outcome cells sum to 140 while the Attempted column lists 137. These discrepancies change the headline acceptance rate from 372/817 = 45.5% to 372/789 = 47.2% and mean the current counts do not support the reported acceptance statistics. Please reconcile the denominator and all failure-mode counts, and identify which table or text is authoritative.","section":"Section 4.1 / Table 14"},{"comment":"The per-hotspot validation oracle for the 330 accepted real-world hotspots is not specified. Table 3 defines oracles for the nine detailed kernels and for the NPB/BOTS suites, and Table 5 gives workload command fragments, but no table or statement maps each of the remaining real-world hotspots to a specific executable check, input asset, golden baseline, or oracle. If a hotspot is accepted under a workload command whose output path does not exercise it, the workload-specific check is vacuous and acceptance collapses to compilation plus positive speedup. The Section 5 caveat that acceptance is not proof of race freedom does not repair the missing mapping. Please provide, for every accepted real-world hotspot, the workload command, input, oracle, and tolerance used, or explicitly restrict the accepted-set claims to the nine kernels for which oracles are defined.","section":"Section 3.5 / Table 3"},{"comment":"The headline token-cost and speedup-improvement figures for the nine detailed kernels are not directly supported by the reported tables. The Claude Code baseline averages 111.1k tokens, while the RepoOMP variants average 58.7k, 50.6k, and 66.4k tokens, giving reductions of about 47%, 54%, and 40%, not 47–68%. Similarly, the abstract's 18–28% speedup improvement range does not match the reported means: 4.94x/4.17x is 18% and 5.34x/4.17x is 28%, but 5.46x/4.17x is 31%. Please state the exact computation for these ranges or correct the abstract and Section 4.2.","section":"Abstract / Section 4.2 / Tables 10–11"}],"minor_comments":[{"comment":"The sentence 'For the 712 attempted real-world transformations, Table 16 reports...' appears to refer to the hotspot funnel, which is Table 14; Table 16 is the ablation funnel. Please fix the cross-reference.","section":"Section 4.1"},{"comment":"The No-Router and No-STC ablation variants are shown only graphically; please provide numeric funnel rows for these variants, as is done for Flat-Ctx, Agent-Only, and Rule-Only in Table 16.","section":"Section 4.6.1 / Figures 3–4"},{"comment":"The tolerance cells use the formatting '10 −5' and '10 −4'; please format these as 10^-5 and 10^-4 for readability.","section":"Table 3"},{"comment":"The STC construction uses LLM-generated summaries of the relevant MAP subgraph; no quality control or fallback for incorrect summaries is described. A sentence on how summary errors are detected or bounded would clarify the verification story.","section":"Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a systems/software-engineering venue and the architecture is plausible. The main risk is reproducibility: the funnel inconsistencies and the missing per-hotspot oracle mapping affect the paper's central acceptance statistics. These are fixable with a supplementary artifact and a corrected table, so I do not recommend rejection. Please ensure the authors provide a complete per-hotspot validation mapping and reconcile counts before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"RepoOMP is a genuinely useful system. It recovers repository-level dependency evidence before generating OpenMP pragmas, routes simple loops to deterministic rules and harder ones to an LLM, and compresses context to the facts that matter. The evaluation is broad and more transparent than most in this area: 951 profiled hotspots across NPB, BOTS, FFmpeg, NCNN, and GROMACS, 372 accepted under compile/workload/speedup checks, with the funnel, accepted-set distributions, repeated-run spreads, ablations, and explicit limitations all disclosed. The token-cost reduction (47–68% vs the unstructured Claude Code baseline) and the stability evidence (CV under 1% on all nine detailed kernels) are real and worth building on.\n\nThe main soft spot is arithmetic. Section 4.1 says 817 hotspots entered attempted transformation, but Table 14 sums to 789. RQ2 says 212 no-speedup rollbacks, but the table sums to 184. The GROMACS row also disagrees with itself: outcome counts sum to 140 while attempted is listed as 137. These discrepancies change the headline acceptance rate from 45.5% to 47.2%. They are likely transcription errors, but they need to be reconciled before the counts can be trusted.\n\nSecond, the per-hotspot validation oracle for the 330 accepted real-world hotspots is not specified. Table 3 gives detailed oracles for the nine case-study workloads and for NPB/BOTS, but no table maps each accepted hotspot to a specific executable check, input asset, or golden baseline. The paper's Section 5 disclaimer is honest, but it does not replace a missing per-hotspot mapping. If a hotspot was accepted under a workload command whose output path does not exercise it, the check is vacuous.\n\nThe nine detailed kernels are deliberately selected; the paper discloses this and gives within-repository ranks, which mitigates the concern. The AutoPar/Polly baselines run under a different task interface, so the comparison is more about protocol suitability than transformation quality; the paper is explicit about this too. The baseline speedup below 1.0 in Table 10 is not a flaw, since the acceptance rule applies only to RepoOMP's own outputs.\n\nOverall, the design is credible and the limitations are stated rather than hidden. The count inconsistencies and the missing per-hotspot oracle mapping are genuine reproducibility soft spots. This paper deserves a serious referee; I would send it to review with a request for a full artifact and reconciled funnel numbers.","headline":"RepoOMP is a genuinely useful hybrid system with a transparent evaluation, but the funnel counts don't reconcile and the per-hotspot validation trail is incomplete; worth a serious referee once the numbers are fixed.","tokens_in":27297,"tokens_out":6091,"would_cite":true,"duration_ms":46991,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RepoOMP recovers hidden dependency evidence before asking an LLM to add OpenMP, accepting 372 of 951 profiled hotspots with a median 2.25x speedup and roughly half the token cost of an unstructured agent.","keywords":["OpenMP parallelization","hotspot optimization","dependency recovery","LLM code generation","repository-scale program analysis","structured context","hybrid rule-agent system"],"falsifier":"Run the nine accepted real-world kernels under ThreadSanitizer with a much larger thread count and multiple distinct inputs while exercising the same output checks: if any of the 330 accepted transformations exhibits an unsynchronized read-write race that changes program output on an input or schedule not covered by the original validation suite, the acceptance oracle is insufficient and the reported safety statistics overstate correctness. A cheaper partial check is to count, for each accepted patch, the number of TSan warnings that are introduced by the patch itself rather than pre-existing in the original code.","tokens_in":26190,"feed_emoji":"⚡","tokens_out":3795,"duration_ms":29945,"temperature":0.7,"pith_summary":"The paper argues that the reason OpenMP auto-parallelization fails on mature repositories is not a lack of transformation skill but a lack of evidence: whether a loop is safe and profitable to parallelize usually depends on non-local facts such as transitive callees, file-scope variables, and shared-state writes. RepoOMP therefore inserts an explicit evidence-recovery stage before generation, builds a repository-wide dependency-and-performance graph, and routes each hotspot to deterministic rules, an LLM agent, or a conservative skip based on propagated risk. The claimed payoff is a system that parallelizes far more hotspots than rule-based tools while being far more stable and token-efficient than unstructured LLM agents. On 951 profiled hotspots across NPB, BOTS, FFmpeg, NCNN, and GROMACS, the paper reports that 372 pass compilation, workload-specific checks, and positive speedup, including 330 real-world hotspots with median 2.25x speedup, and that on nine detailed real-world kernels it beats the Claude Code baseline by 18--28% in speedup while cutting agent-side token cost by 47--68%.","feed_headline":"Hidden dependencies decide OpenMP safety; RepoOMP finds them first","feed_subtitle":"A dependency graph plus bounded prompts lets an LLM parallelize 330 real hotspots at 2.25x median speedup.","key_machinery":"The central object is the Multi-granularity Attributes Performance graph (MAP), a directed property graph with repository, file, and function nodes and control-flow plus data-dependency edges, augmented with runtime hotspot attributes and shared-state access summaries. Its job is to make the non-local evidence that governs parallel safety explicit before generation: it records which callees write file-scope state, which globals a loop touches, and which transitive blockers like I/O or serial control constrain a candidate. The Rule-Agent Router then propagates these blocker attributes bottom-up along the call graph so that a caller inherits the risk of its descendants, and the Structured Transformation Context (STC) serializes the target span, visible shared-state definitions, callee summaries, and a constraint block into a fixed five-field prompt template so the LLM reasons over dependency facts rather than raw repository text.","core_discovery":"The central claim is that hotspot-oriented OpenMP parallelization in repository contexts is best treated as an evidence-recovery problem rather than a code-generation problem: the decisive dependency facts usually live outside the loop being transformed, so the system that survives is the one that recovers, propagates, and compresses those facts before any directive is written. RepoOMP operationalizes this claim with a three-stage pipeline: a Multi-granularity Attributes Performance graph (MAP) records hotspot timing, call relations, and shared-state access summaries; a Rule-Agent Router propagates blocker attributes bottom-up along the call graph and routes each candidate to a deterministic rule engine, an LLM agent with a bounded Structured Transformation Context (STC), or a conservative skip; and a verification loop accepts only transformations that compile, pass workload-specific executable checks, and show positive speedup. The paper reports that this workflow accepts 372 of 951 profiled hotspots, of which 330 come from FFmpeg, NCNN, and GROMACS, with median 2.25x speedup, and that on the nine detailed accepted real-world kernels RepoOMP achieves a mean 5.25x speedup with zero build-failure/wrong-answer cells while the unstructured Claude Code baseline incurs 26 such cells.","pith_inferences":["Editorial inference: the decisive comparison the paper does not run is an oracle-targeted one where a human hands each tool the exact loop to parallelize; the reported rule-based baselines are evaluated on the harder command-level interface, so the 8x-9x versus 1.4x-1.5x gap partly reflects task interface rather than raw transformation ability.","Editorial inference: a natural stress test of the paper's thesis would be to feed RepoOMP's STC to the unstructured baseline and vice versa; if the evidence package alone transfers the gain, the paper's mechanism is confirmed, whereas if the agent still fails, routing and verification play a larger role than claimed.","Editorial inference: the bounded-context design suggests a scaling law opposite to common agent practice: for dependency-sensitive code transformations, more retrieved context should degrade correctness before it improves it, so the optimal policy is to compress to facts rather than enlarge windows.","Editorial inference: the reported ThreadSanitizer reductions (21--49 percent fewer warnings) suggest the STC may also serve as a weak race-prevention filter, a property that could be tested independently by counting newly introduced TSan warnings per transformation before acceptance."],"forward_implications":["If acceptance under compilation, workload checks, and positive speedup is trusted, RepoOMP roughly triples the number of parallelizable hotspots recovered compared with the rule-only branch while eliminating the build-failure and wrong-answer outcomes that dominate the unstructured agent baseline.","The hybrid routing means deterministic rules remain the cheap high-confidence path, so repositories with many regular reduction-style loops should see low token cost and high acceptance simultaneously.","The token waste of the unstructured agent is largely retry churn: the paper reports cases where the baseline spends millions of tokens and still ends in build failure or wrong answer, whereas RepoOMP keeps agent-side usage stable across three different LLM backbones.","The MAP abstraction is not OpenMP-specific, so the same evidence-recovery workflow could be reused to gate other risky whole-repository transformations that depend on cross-file dependency facts.","Because the accepted-set distribution is disclosed rather than cherry-picked, the median 2.25x across 330 real-world hotspots indicates the approach does not depend on a few spectacular kernels."],"supporting_citations":[{"why":"Supplies the classical polyhedral auto-parallelizer baseline whose effectiveness degrades when dependence evidence escapes the local loop region.","marker":"Bondhugula et al. [2008]"},{"why":"Provides Cetus as the conservative source-to-source baseline whose dependence-driven parallelizers reject transformations when aliasing and transitive side effects cannot be discharged statically.","marker":"Dave et al. [2009]"},{"why":"Provides AutoPar/Rose as the rule-based baseline whose local-provability assumption limits recall in repository contexts.","marker":"Quinlan and Liao [2011]"},{"why":"Supplies the repository-level agent failure pattern (incomplete context retrieval) that motivates the routing design, via AutoCodeRover.","marker":"Zhang et al. [2024]"},{"why":"Supports the claim that graph-backed structured retrieval improves large-code generation, motivating MAP as structured evidence rather than raw text.","marker":"Liu et al. [2024]"},{"why":"Defines the Claude Code unstructured repository-agent baseline against which real-world speedup and token cost are compared.","marker":"Anthropic [2025]"},{"why":"Supplies the evidence that repository-level generation depends on restoring structured evidence rather than supplying more raw context, motivating STC.","marker":"Wang et al. [2025]"},{"why":"Supports the executable-validation loop by showing that textual plausibility alone does not guarantee buildability or semantic preservation.","marker":"Yang et al. [2026]"}],"fun_headline_variants":["Evidence-first OpenMP: RepoOMP finds hidden dependencies first","330 real hotspots parallelized via dependency-aware context","LLM parallelizes 330 hotspots at 2.25x median speedup","Rules+LLM: RepoOMP accepts 330 real hotspots"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that passing compilation, workload-specific executable checks, and positive speedup on the tested inputs is sufficient evidence that a transformation is safe and semantically preserved, while the paper itself concedes in Section 5 that acceptance is not a proof of race freedom, schedule-independent correctness, or exhaustive preservation.","fun_headline_variants_meta":{"raw":{"variants":["Evidence-first OpenMP: RepoOMP finds hidden dependencies first","330 real hotspots parallelized via dependency-aware context","LLM parallelizes 330 hotspots at 2.25x median speedup","Rules+LLM: RepoOMP accepts 330 real hotspots"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000581,"raw_usage":{"total_tokens":2823,"prompt_tokens":1123,"completion_tokens":1700,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":739,"completion_tokens_details":{"reasoning_tokens":1625}},"tokens_in":739,"tokens_out":1700,"duration_ms":10623,"temperature":1.0,"reasoning_tokens":1625,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T22:17:28.221940+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the nine accepted real-world kernels under ThreadSanitizer with a much larger thread count and multiple distinct inputs while exercising the same output checks: if any of the 330 accepted transformations exhibits an unsynchronized read-write race that changes program output on an input or schedule not covered by the original validation suite, the acceptance oracle is insufficient and the reported safety statistics overstate correctness. A cheaper partial check is to count, for each accepted patch, the number of TSan warnings that are introduced by the patch itself rather than pre-existing in the original code.","supporting_citations":[],"review_version":1}