{"id":"a862fd19-dd13-4aab-a3f7-90919987e1cc","arxiv_id":"2607.20353","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"MoST improves LLM-guided code optimization by clustering optimization strategies from heterogeneous knowledge sources and transferring them across programming languages.","lead":"This paper presents MoST, a system that mines code-optimization strategies from multiple sources—including an Intel optimization manual—and transfers them across programming languages to guide LLMs in rewriting slow code. On historical optimization tasks and 15 real-world projects, MoST produces more developer-matching patches and larger performance gains than the prior SemOpt approach.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Real-world comparison uses filtered best-of-many composition for MoST/SemOpt but single-patch evaluation for Codex; claimed Codex superiority may be an artifact.","rationale":"The reader's verdict is CONDITIONAL, and I agree with that verdict, but for a different primary reason. The self-balanced clustering assumption (Section III-B) is a valid secondary concern, but it is not the most load-bearing: the ablation study shows weighted clustering contributes a relatively modest 8.97-10.71% improvement in EM, and the pilot selection provides some empirical support for the chosen hyperparameters. In contrast, the real-world project evaluation protocol in Section V-A5 introduces a direct asymmetry between MoST/SemOpt and Codex Agent. MoST and SemOpt are allowed to generate many candidate variants per hotspot, filter out those that do not improve by >5% or that degrade any data point by >2%, and then compose the best eligible variant per hotspot into a final project. Codex Agent is evaluated as a single patch without this filtering or composition. This makes the headline claim that MoST \"significantly outperforms ... Codex\" on real-world projects potentially an artifact of comparing a best-of-many, filtered result against a one-shot patch. The reader's verdict remains CONDITIONAL: the historical reproduction results (RQ1) are unaffected by this protocol asymmetry, and the comparison with SemOpt uses the same protocol, so the central multi-source/cross-scenario claim may still hold; however, the real-world Codex comparison needs to be re-run under a symmetric protocol or the claims should be tempered. A single concrete check — applying the same filtering/composition protocol to Codex — would settle whether this concern lands. Hence no change to the reader's CONDITIONAL verdict is needed, but the rationale for conditionality should emphasize the protocol asymmetry rather than the clustering assumption.","tokens_in":18839,"tokens_out":6374,"duration_ms":52194,"concrete_test":"Apply the identical candidate-generation, eligibility-filtering, and composition protocol to Codex Agent: for each project, allow Codex to generate one candidate patch per hotspot function (or multiple independent patch runs), keep only candidates that improve at least one data point by >5% and degrade no data point by >2%, compose the best eligible candidate per hotspot into a final project version, then compute Max/Avg across all data points. If Codex's composed Max/Avg approaches or exceeds MoST's on several projects (e.g., Flask, Click, image), the claimed real-world superiority over Codex is an artifact of the asymmetric protocol; if Codex remains far below under the same protocol, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the asymmetry in the real-world evaluation protocol (Section V-A5). For MoST and SemOpt, the reported Max/Avg are computed after a candidate-filtering and composition stage: only variants that improve at least one data point by >5% and degrade no data point by >2% are kept; the best eligible variant per hotspot is assembled into the final project version. Codex Agent, by contrast, contributes a single patch that is evaluated as-is (V-A5: \"Codex Agent's project patch is evaluated as that result\"). Thus MoST's Max/Avg are best-of-many per-hotspot selections, while Codex's are one-shot. The abstract's claim that MoST \"significantly outperforms ... Codex\" on 15 real-world projects rests on comparing a composed, filtered result against an unfiltered single patch. The huge relative improvements over Codex (up to 27,288.89% in Avg) are consistent with this asymmetry. The reader's concern about self-balanced clustering (Section III-B, Eq. 3-6) is less load-bearing: even if the selected hyperparameters are suboptimal, the ablation in Table III shows weighted clustering contributes only ~9-11% (50 vs 56 EM) and the pilot selection has some supporting evidence. The evaluation-protocol asymmetry directly affects the headline real-world claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MoST, an LLM-based code optimization framework that converts heterogeneous knowledge sources (historical commits and an Intel optimization manual) into uniform evidence objects, clusters them across sources and scenarios using a self-balanced weighted clustering algorithm, transfers examples to target scenarios when necessary, and generates validated Semgrep rules to guide LLM optimization. The evaluation has two parts: reproduction of 351 historical optimization tasks in C/C++, Python, and Rust, and optimization of 15 real-world projects. The authors report that MoST exceeds SemOpt on historical-task EM/SemEqv by 24.44%–180.00% and 21.88%–37.50% respectively, and that on real-world projects it achieves larger Max/Avg performance improvements than SemOpt and Codex, with ablations and a provenance analysis supporting the design choices.","tokens_in":19217,"tokens_out":10281,"duration_ms":84379,"significance":"MoST addresses two genuine limitations of prior strategy-guided optimizers: it broadens the knowledge sources beyond commits and enables cross-scenario (e.g., cross-language) strategy transfer. The evaluation is broad and mostly careful: three languages, a new Rust benchmark, three baselines, an ablation of both core components, a pilot-based parameter selection, statistical significance checks for performance measurements, and available artifacts. If the claimed results hold, the work is a solid advance for LLM-based code optimization. My main reservation is that the headline comparison with Codex is based on an asymmetric evaluation protocol; a second, lesser concern is that the clustering hyperparameters are selected on the same corpus used for evaluation. Both are addressable and do not, in my view, invalidate the core ideas.","major_comments":[{"comment":"The real-world comparison between MoST/SemOpt and Codex Agent is asymmetric. For MoST and SemOpt, the reported Max/Avg and #≥5%/#≥10% are computed from a final project version assembled by (i) generating candidate optimizations for hotspot functions, (ii) retaining only candidates that improve at least one data point by >5% and degrade none by >2%, and (iii) for each hotspot, keeping the eligible variant with the highest total improvement. Codex Agent, however, contributes a single project patch that is evaluated as that result ('Codex Agent's project patch is evaluated as that result'). Thus the comparison is best-of-many (filtered and composed) versus one-shot. The enormous relative Avg improvements over Codex (up to 27,288.89% per the RQ4 answer) are consistent with this asymmetry. To support the claim that MoST 'significantly outperforms Codex,' the authors must run Codex through the","section":"V-A5, Table IV, RQ4"},{"comment":"The self-balanced clustering parameters (w_d=3, τ=0.76, s_min=5) are chosen by minimizing the balance loss L(θ) on the full evidence corpus, and the historical evaluation tasks are drawn from that same corpus (with exact commit/code matches excluded). The 45-task pilot set is disjoint from the formal benchmark and provides supporting evidence that the criterion tracks EM, but the pilot is small and the parameter search is still on the same data distribution as the evaluation. Because the ablation in Table III attributes only 6 EM points (50 vs 56) to weighted clustering, the overall comparative claims are unlikely to be overturned by a different θ; nevertheless, the paper should report a sensitivity analysis over the configurations in Table II for the main benchmark (or at least for a subset), to demonstrate that the reported improvements are not tied to this particular selection.","section":"III-B (Eqs. 3–6), IV-C"},{"comment":"Success is defined as 'at least one of three runs' satisfying EM/SemEqv. All LLM experiments are run at temperature 0, so the source of run-to-run variation should be clarified; if the three runs are not truly independent, the definition is ambiguous. Even if this protocol is inherited from SemOpt, the reported relative improvements are computed from counts that can be sensitive to a single run (e.g., small-count settings such as Rust with 14 vs 9 successes). I recommend reporting the distribution (e.g., mean±std over the three runs, or majority success) and the per-run rates, so that the stability of the claimed percentages can be assessed. This does not necessarily bias the comparison against baselines, but it is load-bearing for the precise effect sizes in the abstract.","section":"IV-A5, Fig. 3"}],"minor_comments":[{"comment":"The fallback for Rust SemOpt when no candidate passes the >5% filter should be described more precisely; if SemOpt is not given the same composition opportunities as MoST in such cases, the Rust comparison may still favor MoST.","section":"V-A2"},{"comment":"The manual applicability evaluation (89.70%) should report how many suggestions were inspected, the criteria for 'valuable', and inter-rater agreement; comparing to SemOpt's 89.86% with a different LLM (DeepSeek-V3) is not apples-to-apples.","section":"IV-D"},{"comment":"Only 189 document-derived evidence objects are used versus 48,440 commit-derived objects; the claim that documentation contributes 45.73% of effective C/C++ results should be accompanied by counts/confidence intervals, since the absolute numbers are small.","section":"III-A, Table V"},{"comment":"RQ5 counts multiple effective results from the same hotspot/rule; please clarify whether the counts are at the level of data points and report unique hotspot/rule counts as a sanity check.","section":"V-C"},{"comment":"The data-leakage mitigation excludes only exact commit/code matches; near-duplicate versions of the same optimization could still appear in the strategy library. A similarity-based deduplication or an additional leakage analysis would strengthen the threat-to-validity discussion.","section":"VI"},{"comment":"Some entries are hard to parse because spaces are missing between Avg and #≥5%/#≥10% columns (e.g., RocksDB). Please format the table so each column is clearly separated.","section":"Table IV"},{"comment":"Typos: 'otherknowledge' in the abstract should be 'other knowledge'; reference [55] URL contains a space in 'line profiler'; 'super parameters' in Section II-C should be 'hyperparameters'.","section":"Abstract and references"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The protocol asymmetry in the Codex comparison is the main obstacle; if the authors can rerun that comparison under a matched protocol or carefully restrict the claim, I would view the paper favorably. The reliance on the authors' own SemOpt system and benchmark is acceptable given artifact availability, though independent validation of the Rust reimplementation would be desirable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe paper to know: MoST extends strategy-guided LLM code optimization beyond commit corpora to heterogeneous sources (Intel optimization docs, etc.) and across programming-language scenarios, using evidence objects, self-balanced weighted clustering, and example transfer with rule validation. That is genuinely new and it works: on the historical reproduction benchmark, MoST beats SemOpt by 24–180% in exact match and 22–38% in semantic equivalence, with the biggest gains in Rust, where training data is scarce. The evaluation is broad—351 tasks, 15 projects—and artifacts are linked. The ablation shows the weighted clustering step is a modest contributor (~11% EM), while the example transfer/rule validation step is the big one (43–50% drop when removed). The self-balanced clustering criterion is not the load-bearing risk the abstract makes it look like; the pilot validation gives some support and the ablation shows the weights are a small part of the gain.\n\nThe real soft spot is the real-world Codex comparison. For MoST and SemOpt, the reported Max/Avg are computed after filtering candidates (improve ≥5% on one data point, degrade ≤2% on others) and composing the best eligible variant per hotspot into the final project version. Codex Agent contributes a single patch, evaluated as-is. Comparing a composed, filtered best-of-many result against a one-shot patch is not apples-to-apples; the huge relative improvements over Codex—up to 27,288.89% in Avg—are largely an artifact of that asymmetry. The abstract's \"significantly outperforming ... Codex\" claim rests on this. The fix is straightforward: apply the same candidate-filtering and composition protocol to Codex, or report an uncomposed comparison, or soften the claim. The abstract also overstates: MoST's average is slightly lower than SemOpt's on LevelDB, and Codex Spark beats MoST in threshold counts on Flask; the body acknowledges these, the abstract does not.\n\nMinor: success counts \"at least one of three runs\" is optimistic but applied equally; project-level Max/Avg exclude candidates that fail the eligibility filter, which can hide negative results—worth a sensitivity analysis or caveat. Data leakage is mitigated by excluding exact commit/code matches, but LLM training data can't be audited.\n\nBottom line: a serious engineering advance in a growing subfield, worth refereeing. The reproduction experiment is solid; the real-world Codex claim needs rework. A careful referee should insist on a fair Codex protocol or a reduced claim.\n\nRecommendation: send to peer review.","headline":"Solid multi-source extension of SemOpt with real gains on low-resource languages; the Codex comparison in the real-world experiment is apples-to-oranges and needs rework.","tokens_in":19629,"tokens_out":3946,"would_cite":true,"duration_ms":30857,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MoST claims that strategy-guided code optimization improves when strategies are mined from multiple knowledge sources and transferred across languages or architectures.","keywords":["code optimization","large language models","strategy-guided optimization","static analysis rules","cross-language transfer","evidence objects","weighted clustering","program performance"],"falsifier":"Compute the single-scenario cluster counts K_T with a different LLM or after deduplicating descriptions; if the selected hyperparameters change materially and the exact-match and semantic-equivalence results drop by more than the reported margins, the self-balanced selection claim is undermined. Alternatively, run the same benchmark with a rule-validation step that also checks a held-out set of negative examples to see whether the match-before/reject-after check is sufficient to guarantee precision.","tokens_in":1364,"feed_emoji":"⚡","tokens_out":3046,"duration_ms":45084,"temperature":0.7,"pith_summary":"This paper claims that strategy-guided LLM code optimization can be substantially improved by drawing on more than one knowledge source and by letting optimization strategies cross scenario boundaries such as programming language and target architecture. It proposes MoST, which converts items from optimization commits and an optimization manual into uniform evidence objects, clusters them by natural-language description, transfers examples into the target scenario when direct examples are missing, and generates validated static-analysis rules that locate optimizable spots for an LLM. Compared with the prior SemOpt system on 351 historical optimization tasks, MoST reports 24.44%–180.00% more exact-match patches and 21.88%–37.50% more semantically equivalent patches across C/C++, Python, and Rust. On 15 real-world projects it reports higher maximum and average performance improvements than SemOpt and a Codex agent. The sympathetic reading is that multi-source strategy coverage plus cross-scenario rule validation is the mechanism behind these gains.","feed_headline":"Doc and commit strategies lift LLM code optimization by up to 180%","feed_subtitle":"MoST mines optimization tactics from manuals and commits, transfers them across languages, and beats SemOpt on exact-match patches.","key_machinery":"The load-bearing object is the evidence object e=⟨P,E,T,y⟩, a uniform representation of a strategy from any source, consisting of a natural-language description, a before/after code-modification example, applicable scenario tags, and a source-type label. The self-balanced weighted clustering selects the document weight, similarity threshold, and minimum cluster size so that cross-source clustering preserves single-scenario strategy counts, yielding 356 strategy clusters. The example-transfer procedure, with three cases for exact-tag, partial, and absent target-scenario examples, plus the functional rule validation that checks match-before and reject-after behavior, is what makes cross-scenar","core_discovery":"The central claim is that the two coverage limits of strategy-guided optimization—single-source strategy libraries and scenario-bound formalization—can be overcome by representing every piece of optimization knowledge as an evidence object (a natural-language description, a before/after code example, scenario tags, and a source type), clustering across sources with source-weighted density clustering, and transferring examples into the target scenario before generating static-analysis rules. The paper argues that this pipeline lets strategies be discovered from documents as well as commits, and makes them applicable to scenarios where no direct example exists, while the functional rule valida","pith_inferences":["The evidence-object representation likely generalizes to other knowledge sources such as API documentation, compiler optimization guides, or online Q&A; the current implementation ingests only one optimization manual, so the measured documentation contribution may underestimate what a richer multi-source library would yield.","The self-balanced selection criterion is a proxy for downstream quality; a more direct validation against pilot task outcomes, or a reference count robust to duplicate LLM-generated descriptions, might change the chosen hyperparameters.","If the gains are driven mainly by rule-guided localization rather than by the strength of the optimizer model, MoST should remain effective with a smaller or cheaper model—an easily testable extension of the paper's claim.","Cross-scenario transfer could be extended beyond language and architecture to other scenario properties such as library versions, parallelism models, or cache configurations, which the paper's scenario-tag representation already accommodates."],"forward_implications":["Multi-source strategy construction expands coverage beyond historical commits; documentation information appears in 45.73% of effective C/C++ optimization results even though it forms a tiny fraction of the evidence.","Cross-scenario example transfer matters most for low-resource scenarios: 75.53% of effective Python results and 90.91% of effective Rust results came from post-transfer target-scenario candidates.","Weighted clustering protects high-quality, low-frequency sources; removing it reduces successful optimizations by 8.97%–10.71% in the ablation.","Reliable rule generation is critical: disabling example transfer and validation reduces EM by 43.59%–50.00%.","On 15 real-world projects, MoST reports maximum performance improvements of 19.72%–717.42% and average improvements of 4.44%–258.17%, outperforming SemOpt and Codex agents."],"fun_headline_variants":["MoST blends docs and commits to boost code optimization by 180%","Cross-source strategy mining lifts LLM code patches by up to 180%","From manuals to commits: MoST transfers optimization tactics across languages","MoST: multi-source strategy guidance for LLM code optimization gains 180%","Mining optimization strategies from docs and commits boosts LLM patch quality"],"cache_read_input_tokens":20992,"weakest_assumption_plain":"The paper assumes that the single-scenario strategy-cluster counts used as the reference for hyperparameter selection are a correct, noise-free ground truth; if those counts are unreliable because of duplicate or noisy LLM-generated descriptions, the chosen weights would preserve that noise and degrade every downstream step.","fun_headline_variants_meta":{"raw":{"variants":["MoST blends docs and commits to boost code optimization by 180%","Cross-source strategy mining lifts LLM code patches by up to 180%","From manuals to commits: MoST transfers optimization tactics across languages","MoST: multi-source strategy guidance for LLM code optimization gains 180%","Mining optimization strategies from docs and commits boosts LLM patch quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000828,"raw_usage":{"total_tokens":3500,"prompt_tokens":835,"completion_tokens":2665,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":2569}},"tokens_in":579,"tokens_out":2665,"duration_ms":15218,"temperature":1.0,"reasoning_tokens":2569,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T10:02:36.441830+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the single-scenario cluster counts K_T with a different LLM or after deduplicating descriptions; if the selected hyperparameters change materially and the exact-match and semantic-equivalence results drop by more than the reported margins, the self-balanced selection claim is undermined. Alternatively, run the same benchmark with a rule-validation step that also checks a held-out set of negative examples to see whether the match-before/reject-after check is sufficient to guarantee precision.","supporting_citations":[],"review_version":1}