{"id":"6a0b37c9-6aab-4a2d-864f-ad38ed534cc3","arxiv_id":"2608.08352","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SimP combines Perses-style deletion with a long-tail detector and LLM-based semantic and syntactic mutation to reduce compiler bug test cases 1.75x faster than the better of Perses and LPR, while keeping reduced programs comparable in size.","lead":"SimP is a hybrid program reducer that uses traditional syntax-guided deletion first and then switches to LLM-based semantic and syntactic reduction once progress slows. It reports 1.75x faster reduction on average than two prior reducers, comparable final program sizes, and under $0.50 in LLM cost per bug test case.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 1.75x speedup is measured on the same 23 benchmarks used to set the long-tail detector thresholds and DSPy-optimized prompts, with no held-out validation or error bars; the central efficiency claim may be tuned to the evaluation set.","rationale":"The reader's conditional verdict is appropriate. The strongest evidence is the consistent left-shifted CDF in Figure 9 and the component ablation, which show the design can reduce time without sacrificing size on these benchmarks. What the evidence does not yet show is that the quantitative gain transfers. The two most load-bearing choices—when to switch to the LLM and what prompt to use—have tunable parameters that are not varied or validated out-of-sample, and the evaluation appears to be a single pass over the development set. I do not see an internal inconsistency: the detector formula in §4.1.1 is coherent, the bug-preservation filtering in §4.2.3/§4.3.2 is a sound oracle for reducing false positives, and the monetary cost is independently plausible. The concern is external validity of the headline speedup, not correctness of the pipeline. A successful held-out evaluation would remove the concern; a failed one would reduce the paper to a plausible engineering recipe with unverified quantitative claims. This matches the reader's weakest_assumption closely, so I agree with it and would keep the CONDITIONAL verdict rather than accepting the numbers at face value.","tokens_in":16201,"tokens_out":6367,"duration_ms":62286,"concrete_test":"Fix the released artifact's detector thresholds and DSPy prompt template. Collect a fresh held-out set of miscompilation triggers (e.g., 20 recently filed GCC/LLVM bugs not in [1,2]) and run SimP, Perses, and LPR end-to-end on each, with at least 5 repetitions using different LLM temperatures/seeds. Report the paired geometric-mean speedup with a 95% bootstrap confidence interval and also grid-sweep λ/K/p/j on the held-out set. If the CI lower bound falls below 1.0, or if the speedup drops below roughly 1.2x, the claimed efficiency gain is not separable from benchmark-specific tuning and stochastic noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in §5.2.1 is a 1.75× geometric-mean speedup over the per-benchmark better of Perses and LPR. That speedup is produced by the long-tail detector in §4.1.1 (thresholds λ, K, p, window fraction j) and by the DSPy-optimized prompts in §4.2.2, which control when and how the LLM stages take over. The paper reports that these parameters are in appendix §C and that the prompt is optimized with DSPy, but it does not report any held-out benchmark split, any sensitivity analysis of the thresholds, or repeated runs with confidence intervals. The 23 benchmarks in §5.1 are the same collection used in development, so both the detector thresholds and the prompt templates may be tuned to the long-tail shapes and bug schemas of exactly the programs on which the speedup is measured. Since the detector/prompt pair is the mechanism that creates the efficiency gain, this makes the numeric claim externally fragile even though the hybrid architecture is plausible and the ablation supports each component's contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SimP is a hybrid program reduction framework that combines traditional syntax-guided deletion (based on Perses) with two LLM-driven components: a semantic-guided reducer that synthesizes smaller bug-preserving programs, and a syntax-guided mutator that rewrites the program into a form more amenable to further deletion. A runtime detector (§4.1.1) identifies the long-tail phase of reduction and triggers the switch to the LLM stages. The paper evaluates SimP on 23 real-world miscompilation benchmarks, reporting a 1.75x geometric-mean speedup over the per-benchmark better of Perses and LPR, comparable final reduction sizes, and less than $0.50 LLM cost per benchmark. An ablation study attributes the gains to both the hybrid architecture and the structured prompt design.","tokens_in":16417,"tokens_out":5031,"duration_ms":41593,"significance":"If the reported results hold, SimP addresses a practically important bottleneck: syntax-guided reducers spend most of their runtime in a long tail of diminishing returns. The architectural insight—combining cheap rule-based deletion with LLM-based coordinated edits—is well motivated by the three observations in §3, and the ablation study provides evidence that each component contributes. The paper also ships an artifact and uses public benchmarks, which supports reproducibility. The main limitation is that the quantitative speedup is measured on the same benchmarks used to tune the detector thresholds and prompts, and the evaluation lacks sensitivity analysis and error bars; until that is addressed, the exact speedup numbers should be treated as indicative rather than definitive.","major_comments":[{"comment":"The central efficiency claim—a 1.75x geometric-mean speedup—is measured on the same 23 benchmarks used to set the long-tail detector thresholds (λ, K, p, j) and the DSPy-optimized prompt templates. The paper does not report a held-out benchmark split, a sensitivity analysis over these parameters, or repeated runs with confidence intervals. Because the detector and prompt pair determines when and how the LLM stages take over, the numeric speedup may be partly tuned to this specific suite and could be externally fragile. Please provide either a sensitivity analysis showing stable performance across plausible parameter choices, a holdout evaluation on additional benchmarks, or a clear argument for why the current parameters are not overfit to the evaluation set.","section":"§5.2.1, §4.1.1, §4.2.2"},{"comment":"The formula for c_t is ambiguous: the numerator 'token_t(1−j%) − token_t' can be read as token_t multiplied by (1−j%) rather than as the token count at time t(1−j%) minus the token count at time t. Since this quantity is load-bearing for the long-tail detection, please clarify the notation and define the time-window endpoints precisely.","section":"§4.1.1"},{"comment":"The ablation conclusions that 'every component contributes measurably' are based on single runs per variant. Given the stochastic nature of LLM sampling, the reported differences (e.g., 50% smaller with the mutator, 47% larger without the reducer, 41% larger without Perses) should be accompanied by confidence intervals or repeated runs with different random seeds to rule out run-to-run variation.","section":"§5.3"},{"comment":"The claim that the size gap on gcc116906 (260 vs. 115 tokens) 'does not noticeably increase the difficulty of human inspection' is supported only by a single qualitative example. If the paper wishes to maintain this assertion as part of the 'comparable reduction quality' claim, it needs a more systematic evaluation—for example, multiple examples or a small human study—rather than one anecdotal instance.","section":"§5.2.2"}],"minor_comments":[{"comment":"There is a typo in 'thesyntax-guided program reductioncategory'; also the sentence 'All these tools belong to the syntax-guided program reduction category' appears immediately after describing LPR, which is confusing because the preceding sentences discuss C-Reduce and ddSMT as well.","section":"§1"},{"comment":"The caption says 'We highlight in the code snippet that is mutated from the input program,' which is grammatically awkward; please rephrase to clearly state which parts are highlighted, and describe the green/red color coding in a way that remains legible in grayscale printing.","section":"Figure 7 caption"},{"comment":"Ye Xiong and Xiangyu Gao are listed with the same email address (xiangyug@cs.washington.edu); please correct this likely typographical error.","section":"Author affiliations"},{"comment":"The evaluation uses 'GPT-5.1' but does not specify the exact model identifier, access date, or inference API; please provide this information to support reproducibility.","section":"§5.1"},{"comment":"The statement that 'the per-benchmark cost difference is ≤$0.3' would benefit from having the per-benchmark costs tabulated, for example in the appendix table, so that the average is not the only reported statistic.","section":"§5.2.3"},{"comment":"The description of the benchmark provenance is thin: references [1] and [2] are artifact URLs, but the paper does not explain how the 23 benchmarks were selected from those artifacts or which inclusion criteria were applied; a brief sentence would clarify the evaluation scope.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for a programming-languages/systems venue and the artifact is a plus. The main risk is the potential overfitting of the detector thresholds and prompts to the evaluation suite; I would encourage the revision to include sensitivity analysis or a holdout evaluation, and to temper the strength of the speedup claims until then. The other load-bearing issue is the ambiguous c_t formula in §4.1.1."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SimP is a sensible hybrid reducer and probably a real improvement over Perses and LPR, but the central speedup number is only as strong as the tuning story. The novelty is not LLMs in reduction—LPR already does LLM refinement—but the explicit long-tail detector and the two-stage handoff: run Perses until a runtime score says progress has slowed, then alternate between a semantic LLM reducer that synthesizes from contrastive examples and a mutator that canonicalizes syntax for further Perses passes. The ablation isolates each piece and shows each contributes, and the gcc107176 trajectory is a convincing illustration of avoiding the long-tail. The contrastive-example collection from the Perses run is a nice idea: it gives the LLM positive and negative pairs without extra compilation effort, and the per-benchmark cost under $0.5 is reasonable.\n\nThe main weakness is the evaluation's calibration. The long-tail detector thresholds (lambda, K, p, j) and the DSPy-optimized prompts are set on the same 23 benchmarks used for the reported 1.75x speedup. There is no held-out split and no sensitivity analysis, so part of the gain may be fitting to these specific programs. Single-run times with no error bars make the exact geometric mean fragile. I would want repeated runs, threshold sensitivity, and at least one unseen benchmark suite before trusting the quantitative claim. Also, C-Reduce is conspicuously absent from the comparison for C compiler bugs; the authors cite it but do not run it, and the conclusion of comparable reduction quality is only relative to Perses and LPR. That does not sink the size claim, but it is less evidence than the text implies.\n\nNone of this kills the paper. The architecture is plausible and the ablation supports the qualitative story. The 1.75x number is a claim to be validated, not a headline to be trusted as-is. This is a paper for compiler debugging and software engineering researchers. It deserves peer review; I would send it out with a request for a held-out evaluation and repeated runs. My own verdict would be conditional accept: the design is worth publishing, but the central efficiency claim needs more discipline.","headline":"A plausible hybrid reducer whose architecture is well supported by ablation, but the headline 1.75x speedup is tuned on the same benchmarks it is measured on.","tokens_in":16975,"tokens_out":1980,"would_cite":true,"duration_ms":18800,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SimP claims that pairing syntax-guided deletion with LLM-based semantic reduction collapses the long-tail slowdown of program reducers, cutting average reduction time by 1.75x while keeping output size comparable.","keywords":["program reduction","compiler bugs","miscompilation","large language models","syntax-guided reduction","semantic-guided reduction","long-tail detection","hybrid program reduction"],"falsifier":"Take a held-out suite of miscompilation bugs that was not used in any threshold or prompt tuning, run SimP with the appendix's fixed parameters against Perses and LPR, and measure end-to-end time to convergence; if the geometric-mean speedup over the better baseline is near 1x, or SimP often leaves outputs far above 60 lines, the paper's central claim is refuted.","tokens_in":15985,"feed_emoji":"🐛","tokens_out":7044,"duration_ms":59850,"temperature":0.7,"pith_summary":"The paper argues that modern program reducers waste most of their runtime in a long tail: after an initial phase of fast bulk deletion, local AST-level edits stall, and hours are spent removing a handful of tokens. SimP detects that stall at runtime and hands the program to two LLM-based stages, one that reasons about the bug-triggering semantics and synthesizes a smaller program, and one that rewrites the program into a form the deletion reducer can shrink further. On 23 real-world miscompilation benchmarks, SimP finishes with a 1.75x geometric-mean speedup over the faster of Perses and LPR, up to 2.55x, while final programs stay under 60 lines of code and LLM cost stays below $0.50 per benchmark. The core claim is that syntax-guided deletion is fast early and semantic reasoning is what breaks the logjam late, so combining them at the right switch point is faster than either paradigm alone.","feed_headline":"SimP cuts compiler-bug reduction time by 1.75x, up to 2.55x","feed_subtitle":"Switching to LLM reasoning when progress stalls keeps bugs small while saving about an hour per case.","key_machinery":"The central mechanism is the long-tail detector, a runtime score $\\mathit{score}_t = r_t + p\\cdot c_t$, where $r_t = \\mathit{token}_t/\\mathit{token}_0$ is the remaining token ratio and $c_t = (\\mathit{token}_{t(1-j\\%)} - \\mathit{token}_t)/(\\mathit{token}_0 - \\mathit{token}_t)$ is recent reduction progress normalized by total reduction achieved; SimP switches when $\\mathit{token}_t \\le \\lambda$ and $\\mathit{score}_t \\le K$. Around this switch, two LLM stages do the heavy lifting: a semantic reducer that mines contrastive positive and negative examples from the prior Perses run, follows a four-step reasoning guide, and uses best-of-$N$ selection with $N=6$, and a mutator that applies one of five single-strategy rewrite families region by region, filters candidates by bug preservation, and synthesizes surviving mutations back into one program.","core_discovery":"SimP's central discovery is that the long-tail slowdown of syntax-guided reducers is not a fixed cost but a detectable phase transition, and that LLM-driven global edits are the right tool once local AST edits stop paying. With a lightweight runtime detector, the reducer switches from Perses-style deletion to a semantic LLM reducer that produces multiple candidate reductions and keeps the smallest one that still triggers the bug, plus a syntax-guided LLM mutator that canonicalizes declarations, inlines functions, and flattens control flow. The resulting workflow reaches smaller programs in less time than either pure syntax-guided reduction or pure naive LLM reduction, and the ablation study shows that removing any component measurably degrades output quality.","pith_inferences":["Editorial inference: the long-tail detector could serve as a standalone early-stopping signal for existing reducers even without LLMs, handing off to a human or a more expensive search at the moment local edits begin to stall.","Editorial inference: because the switching thresholds and prompt templates were optimized with access to the same 23 benchmarks used in the evaluation, a held-out suite is needed before the 1.75x average speedup can be treated as a general property of the method rather than of the tuned configuration.","Editorial inference: the single-strategy mutation design suggests a testable extension to other structured artifacts, such as SMT formulas, database queries, or configuration files, where local edits also stall and global semantic reasoning may help."],"forward_implications":["Developers can expect hard miscompilation bugs to be reduced in about an hour less per case on average, with the worst cases dropping from over seven hours to roughly 100 minutes.","Final reduced programs remain small enough for direct human inspection, under 60 lines of code on all 23 benchmarks, with no single reducer consistently producing the smallest output.","The monetary cost of LLM calls stays below $0.50 per benchmark, so the hybrid strategy is practical for routine use rather than a research luxury.","Ablations show each component earns its place: removing the semantic reducer enlarges outputs by about 47% on average, removing the mutator by about 50%, and removing Perses by about 41%, so the hybrid interaction rather than any single stage delivers the result."],"supporting_citations":[{"why":"Perses, the syntax-guided deletion reducer SimP builds on for early bulk deletion and cleanup, and one of the two baselines.","marker":"[27]"},{"why":"LPR, the LLM-aided reducer used as the second baseline; SimP's speedup is measured against the per-benchmark better of it and Perses.","marker":"[31]"},{"why":"One of the two benchmark artifact suites that supply the real-world miscompilation cases used in the evaluation.","marker":"[1]"},{"why":"The other benchmark artifact suite supplying the miscompilation cases.","marker":"[2]"},{"why":"The prompt-optimization method used to tune the concrete prompt templates for the LLM stages.","marker":"[15]"},{"why":"Csmith, whose large generated C programs motivate the size problem that program reduction addresses.","marker":"[30]"},{"why":"C-Reduce, an additional syntax-guided reducer whose long-tail trajectory is measured in the characterization study.","marker":"[25]"}],"fun_headline_variants":["SimP: LLM takeover after local edits stall speeds reduction","SimP: Hybrid syntax-semantic reducer cuts time to shrink bugs","SimP: Up to 2.55x faster compiler-bug reduction with LLM assist","SimP: Detect stagnation, switch to LLM, shrink programs faster","SimP: Semantic and syntax reducers combine for efficient minimize"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the long-tail detector thresholds and the prompt templates, tuned with access to the same 23 benchmarks used for evaluation, will keep working on unseen programs rather than being overfit to those benchmarks.","fun_headline_variants_meta":{"raw":{"variants":["SimP: LLM takeover after local edits stall speeds reduction","SimP: Hybrid syntax-semantic reducer cuts time to shrink bugs","SimP: Up to 2.55x faster compiler-bug reduction with LLM assist","SimP: Detect stagnation, switch to LLM, shrink programs faster","SimP: Semantic and syntax reducers combine for efficient minimize"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000323,"raw_usage":{"total_tokens":1755,"prompt_tokens":830,"completion_tokens":925,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":446,"completion_tokens_details":{"reasoning_tokens":829}},"tokens_in":446,"tokens_out":925,"duration_ms":9201,"temperature":1.0,"reasoning_tokens":829,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:07:00.936267+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out suite of miscompilation bugs that was not used in any threshold or prompt tuning, run SimP with the appendix's fixed parameters against Perses and LPR, and measure end-to-end time to convergence; if the geometric-mean speedup over the better baseline is near 1x, or SimP often leaves outputs far above 60 lines, the paper's central claim is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Perses, the syntax-guided deletion reducer SimP builds on for early bulk deletion and cleanup, and one of the two baselines."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LPR, the LLM-aided reducer used as the second baseline; SimP's speedup is measured against the per-benchmark better of it and Perses."},{"cited_title":"CReal Artifact","cited_arxiv_id":null,"evidence_quote":"One of the two benchmark artifact suites that supply the real-world miscompilation cases used in the evaluation."},{"cited_title":"LegoFuzz Artifact","cited_arxiv_id":null,"evidence_quote":"The other benchmark artifact suite supplying the miscompilation cases."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prompt-optimization method used to tune the concrete prompt templates for the LLM stages."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Csmith, whose large generated C programs motivate the size problem that program reduction addresses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"C-Reduce, an additional syntax-guided reducer whose long-tail trajectory is measured in the characterization study."}],"review_version":1}