{"id":"55e5750a-1a6b-4cca-a0e4-4d60d14387aa","arxiv_id":"2607.26805","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Draft-guided, API-and-similarity context selection plus parallel draft verification improves repository-level code generation quality while cutting tokens and latency versus RAG and prior compressors.","lead":"MRCoder filters noisy retrieved code context for repository-level LLM generation using a Map–Reduce pipeline: small-model drafts guide structure-aware selection, then a larger model generates with draft-based parallel verification. It reports higher Pass@1 with 30–50% fewer tokens and up to ~52% less inference time on CoderEval and DevEval.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Quality gains rest on small-model drafts being faithful enough selectors of the contexts the target model actually needs.","rationale":"The reader correctly isolates the load-bearing premise: small-model drafts on sequential partitions must be good enough selectors for the large model. Everything else (Map–Reduce framing, API+BM25 dual signal, speculative verification, multi-K tables, token/time breakdowns) is either standard or secondary. The paper supplies supportive but incomplete evidence—ablations, a single qualitative case, and stable-but-not-dominant gains versus LongCodeZip/RepoFormer—while Threats already flags language, retriever, and model-size limits. No internal contradiction appears; the concern is empirical under-stress-testing of the proxy, not a flaw in the reported numbers themselves. Hence the verdict stays CONDITIONAL (accept-shaped once variance, release artifacts, and a direct proxy-fidelity check are supplied) and I agree with the reader’s weakest-assumption diagnosis. A negative concrete-test outcome would push toward a weaker conditional or rejection of the quality half of the claim; a positive outcome would raise confidence without changing the overall posture.","tokens_in":24010,"tokens_out":637,"duration_ms":30862,"concrete_test":"On the CoderEval Python subset with Qwen2.5-Coder (K=5 and K=10), replace SADGS with an oracle that keeps the single context block whose removal most hurts target-model Pass@1 (leave-one-out); measure how often SADGS retains that block and the resulting Pass@1 delta. If retention <60% or the oracle gap exceeds 4 absolute points, the draft-proxy claim weakens materially.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (higher Pass@1 plus 30–50% fewer tokens / up to ~52% less wall time) requires that drafts produced by a 1.3–1.5 B model on fixed sequential partitions (M=4) of BM25 Top-K are reliable proxies: API-call intersection with the draft’s external calls plus Top-L=1 BM25(draft, context) must retain the fragments that drive correct target-model generation and discard the rest (§3.3.1–3.3.2, Eqs. 5–8). If the small model systematically omits needed repository APIs or latches onto superficial lexical similarity, SADGS can drop critical context or keep noise; the Reduce-phase parallel verification then only accelerates whatever (possibly degraded) context remains. Ablations (Table 3) show both signals help on average, yet occasional non-degradations or small gains when a component is removed, the hyper-parameters chosen only from “preliminary experiments,” and the exclusive use of same-family draft/target pairs leave the proxy assumption under-tested. Efficiency numbers are less fragile because parallel verification is standard speculative decoding, but the joint quality+efficiency headline stands or falls with draft fidelity.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"MRCoder proposes a post-retrieval Map–Reduce context selection framework for repository-level code generation. In the Map phase, Top-K retrieved contexts are partitioned into fixed groups; a lightweight draft LLM generates per-group drafts; Structure-Aware Draft-Guided Selection (SADGS) retains contexts via (i) overlap between draft external API calls and context internal/external APIs (Eq. 5) and (ii) Top-L BM25 similarity between draft and contexts (Eqs. 6–7), merged in Eq. 8. In the Reduce phase, selected contexts feed a larger target LLM, and Map drafts are reused for parallel/speculative verification (Algorithm 1) to cut decoding latency. On CoderEval and DevEval with Qwen2.5-Coder and DeepSeek-Coder pairs, the paper reports higher Pass@1 than RAG and several selection/compression baselines in most settings, with ~30–50% fewer context tokens and up to ~52% lower wall-clock time, plus ablations of API, similarity, and parallel decoding components.","tokens_in":24298,"tokens_out":1627,"duration_ms":43371,"significance":"Repository-level RAG noise and context bloat are genuine, practical bottlenecks; a method that jointly improves Pass@1 and reduces tokens/latency is valuable to the SE and code-LLM communities. Strengths include a clear pipeline, structure-aware selection signals (API calls via tree-sitter plus draft-conditioned BM25), efficiency decomposed into Proc.T/Gen.T/All.T and tokens, ablations (Tables 3–4, Fig. 4), partition/draft-selection discussions (§6.2–6.3), a concrete case study (§6.1), and released code. The contribution is primarily empirical/systems rather than theoretical; if the draft-as-proxy assumption holds more broadly, the Map–Reduce + SADGS pattern is a reusable design point for repo-level generation.","major_comments":[{"comment":"The central quality claim depends on small-model drafts being faithful selectors of contexts the target needs (§3.3.1–3.3.2, Eqs. 5–8). All reported runs use same-family draft/target pairs (Qwen 1.5B→7B, DeepSeek 1.3B→6.7B; §4.4). Table 3 shows that removing API or Sim sometimes does not hurt and occasionally slightly improves Pass@1 (e.g., DeepSeek CoderEval K=3/5 w/o_API). Without cross-family or weaker-draft controls, or analysis of when drafts omit needed APIs vs. latch onto lexical similarity, it remains unclear how much of the Pass@1 gain is robust to draft fidelity rather than favorable pairing and BM25 ranking. A targeted experiment or failure analysis would substantially strengthen the load-bearing proxy assumption.","section":"§3.3.2, §5.3 Table 3, §4.4"},{"comment":"Hyperparameters that define SADGS—group size M=4 and logic Top-L=1—are stated as chosen from “preliminary experiments” (§4.4) without a reported sensitivity sweep in the main results. §6.2 compares partition strategies on Pass@1 but does not systematically vary M or L against both quality and token/time trade-offs. Because Eqs. 5–8 and the sequential first-group draft choice (§3.4.2) are the mechanism behind the headline 30–50% token cut and stable-vs-K behavior, the paper should show that gains are not brittle to these free parameters (or report the preliminary grid).","section":"§4.4, §3.3.1–3.3.2, §6.2"},{"comment":"Table 1’s strongest relative claim (e.g., +52.7% Pass@1 vs RAG for DeepSeek on CoderEval at K=10) coincides with a sharp RAG collapse (23.9). On DevEval with DeepSeek, MRCoder is tied or behind RL-Coder and/or LongCodeZip at several K (e.g., K=3,5,7,10). The narrative of consistent superiority and “more stable” K-behavior is only partly supported. Please qualify the abstract/§5 claims to match the full grid (best-K and average-K, and when compression baselines win), and avoid leading with the most favorable relative delta without that context.","section":"Table 1, §5.1, Abstract"},{"comment":"Algorithm 1 extends standard speculative decoding with rejection fallback plus span re-alignment into the remaining draft suffix. The text asserts that parallel verification preserves the target model’s distribution when the same decode/sampling strategy is used (§3.4.1). Span matching and resuming from an interior draft position can change the accepted prefix path relative to pure autoregressive or vanilla speculative decoding. Please either prove/argue invariance of the final token distribution under temperature 0 (as used), or empirically verify Pass@1 with vs. without PD on the same refined context G* (quality is claimed unaffected; only w/o_PD time is ablated in Fig. 4).","section":"§3.4.1–3.4.2, Algorithm 1, Fig. 4"}],"minor_comments":[{"comment":"Figure 1 caption and axis labels are useful; ensure the main text states the exact retrieval and model setup in the figure itself for standalone reading.","section":"Fig. 1"},{"comment":"Typos/wording: “Phrase Pipline” → “Phase Pipeline” (§3.4.2); “Map Phrase” / “Reduce Phrase” appear in places instead of Phase; “are are effective” (§3.1); “pipline” in §3.4.2.","section":"§3.1, §3.4.2"},{"comment":"ACM reference date shows 2018 and placeholder conference/DOI (XXXXXXX). Update metadata before camera-ready.","section":"Front matter"},{"comment":"DevEval filtering to 1,462/1,825 runnable instances (§4.1) is reasonable; briefly state whether failures are environment vs. test flakiness and whether the filtered subset biases difficulty.","section":"§4.1"},{"comment":"Threats (§7) correctly note Python-only and BM25-centric evaluation. A single dense-retriever or cross-language spot check would help the “retrieval-agnostic” claim, even if left as limitation if resources are tight.","section":"§7"},{"comment":"Report whether Pass@1 differences are averaged over one deterministic run (T=0) only; if so, note variance across seeds is N/A but sensitivity to retrieval ties/order may still matter.","section":"§4.4, §5.1"}],"recommendation":"major_revision","confidential_remarks":"Solid empirical SE systems paper; not a reject. Major revision is mainly to stress-test the draft-proxy assumption and to tone Table 1/abstract claims to the full result grid. No integrity red flags; code link is a plus. Fit is appropriate for a software engineering / empirical AI-for-code venue."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a clean engineering fix for the familiar RAG noise/latency tradeoff in repository-level code generation. Partition Top-K, draft with a small model, keep contexts that share APIs with the draft or look like it under BM25, then generate with the big model while verifying the draft in parallel. On CoderEval and DevEval with Qwen2.5-Coder and DeepSeek-Coder they beat standard RAG and several published selectors on Pass@1 in most cells, cut tokens ~30–50%, and cut wall time up to ~52%.\n\nWhat is actually new is the packaging, not the parts. Map-style partitions, small-model drafts, tree-sitter API sets, BM25, and Leviathan-style parallel verification are all known. SADGS (draft external-API ∩ context APIs, plus Top-L BM25 to the draft) is a sensible dual signal and they wire it into a full Map–Reduce pipeline with measured quality and cost. That packaged result is not already in RepoCoder, RL-Coder, RepoFormer, or LongCodeZip.\n\nThey do the empirical work properly. Multi-K tables, Proc/Gen/All time split, token counts, ablations for API, Sim, and parallel decoding, partition and draft-selection discussions, a readable case study, and an honest threats section. Code link is promised. Circularity is low: selection is guided by drafts, correctness is execution-style Pass@1.\n\nSoft spots, in proportion. The load-bearing assumption is that 1.3–1.5B same-family drafts on fixed sequential groups (M=4, L=1 from “preliminary experiments”) are good enough proxies for what the target needs. Ablations mostly support both signals, but a few cells do not drop when a piece is removed, and they never really stress cross-family drafts, other languages, or non-BM25 retrievers. Efficiency is less fragile—parallel verification is standard speculative decoding—so the joint quality+efficiency headline is only as strong as draft fidelity. Novelty is compositional; significance is product/workflow level inside coding assistants, not a new scientific regime.\n\nWho it is for: people building repo-level RAG or coding-agent context stacks who care about latency and token bills. Worth a serious referee. I would skim the repo and cite the pipeline if I were writing on context selection or speculative decode for code. Bring to reading group only if the group is deep in LLM-for-SE systems; otherwise a quick share is enough.\n\nRecommendation: send to peer review. Accept-shaped contingent on release artifacts, some variance, and clearer limits on the draft-proxy claim—not a desk reject.","headline":"Solid systems paper: draft-guided API+BM25 context filtering plus speculative decode gives real Pass@1 and latency wins on repo code gen; novelty is compositional, not foundational.","tokens_in":24970,"tokens_out":660,"would_cite":true,"duration_ms":19569,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"MRCoder filters retrieved repository context with draft-guided API and logic signals, improving code generation accuracy while cutting tokens 30–50% and inference time by up to 52%.","keywords":["Large Language Models","Context Selection","Repository-Level Code Generation","Efficient Inference","Retrieval-Augmented Generation","Draft-Guided Selection","Parallel Decoding"],"falsifier":"On the same CoderEval/DevEval splits, replace SADGS with random retention of the same token budget (or force-drop contexts that share the draft’s external APIs) and check whether Pass@1 and the reported time/token gains over RAG and LongCodeZip disappear.","tokens_in":24804,"feed_emoji":"💻","tokens_out":960,"duration_ms":27479,"temperature":0.7,"pith_summary":"Repository-level code generation needs project-specific context, but retrieving more snippets often adds noise that hurts correctness and slows inference. This paper argues that the fix is not better retrieval alone, but a post-retrieval Map–Reduce selection step: a small model drafts code on short partitions of the retrieved set, then keeps only contexts that match the draft’s API calls or logical shape; a larger model then generates from the thinned context and verifies draft tokens in parallel. On two standard benchmarks with two code LLM families, that design raises Pass@1 over strong RAG and compression baselines while using far fewer tokens and less wall-clock time. A sympathetic reader cares because the same noisy Top-K problem shows up whenever LLMs must ground generation in a large codebase, and the paper claims a practical way to keep the useful bits without paying the full context cost.","feed_headline":"Draft-guided filters cut repo code tokens 30–50%","feed_subtitle":"Map–Reduce keeps API- and logic-matched snippets, lifting Pass@1 and slashing inference time up to 52%.","key_machinery":"Structure-Aware Draft-Guided Selection (SADGS): after a small draft model generates code on each fixed-size partition of Top-K contexts, keep a context if it shares API calls with the draft’s external calls or ranks in the Top-L BM25 similarity to the draft; merge those sets, then generate with the target LLM under parallel draft-token verification.","core_discovery":"The authors claim that Structure-Aware Draft-Guided Selection (SADGS) inside a Map–Reduce pipeline yields both higher functional correctness and lower cost than standard RAG, RL-tuned retrieval, selective-retrieval control tokens, or perplexity-based line compression. Drafts from a lightweight model on fixed sequential partitions supply reliable signals—API-call overlap and BM25 logical similarity—so the Reduce-phase target model sees concise, multi-perspective context and can accept draft tokens in parallel, producing up to about 53% relative Pass@1 gains and roughly 30–50% fewer tokens with inference time cuts up to about 52% on CoderEval and DevEval.","pith_inferences":["If small-model drafts systematically miss rare project APIs, hybrid signals (static call graphs plus drafts) may be needed before scaling to multilingual or multi-million-line repos.","The same Map–draft–filter pattern could transfer to other long-context grounded tasks where structure (calls, schemas, citations) is cheaper to check than full perplexity sweeps.","Acceptance rates of draft tokens under the target model would be a useful diagnostic of when First-Draft selection helps most."],"forward_implications":["Raising Top-K for recall need not force a quality–latency tradeoff if draft-guided filtering follows retrieval.","API-call consistency plus logical similarity can replace multi-pass perplexity scoring as a cheaper selection signal.","Map-phase drafts can double as speculative tokens for Reduce-phase parallel verification, offsetting draft cost.","Sequential ranking-preserving partitions are enough; clustering or random grouping is not required for the gains.","The pipeline is retrieval-agnostic, so the same selection step can sit on BM25 or other retrievers."],"fun_headline_variants":["MRCoder Map-Reduce drafts cut repo code tokens 30-50%","SADGS draft signals lift Pass@1, slash inference up to 52%","Structure-aware drafts trim RAG context for repo code gen","Lightweight drafts select API-logic snippets, cut tokens half","Map-phase drafts guide Reduce context, boost code accuracy"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"Drafts from a much smaller model on simple sequential partitions of retrieved blocks are trustworthy enough proxies of what the large model needs that API overlap and draft–context similarity will keep the truly useful snippets and drop the rest.","fun_headline_variants_meta":{"raw":{"variants":["MRCoder Map-Reduce drafts cut repo code tokens 30-50%","SADGS draft signals lift Pass@1, slash inference up to 52%","Structure-aware drafts trim RAG context for repo code gen","Lightweight drafts select API-logic snippets, cut tokens half","Map-phase drafts guide Reduce context, boost code accuracy"]},"model":"grok-4.5","effort":"low","cost_usd":0.004409,"raw_usage":{"total_tokens":1410,"prompt_tokens":904,"num_sources_used":0,"completion_tokens":73,"cost_in_usd_ticks":44088000,"prompt_tokens_details":{"text_tokens":904,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":433,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":904,"tokens_out":73,"duration_ms":7143,"temperature":1.0,"reasoning_tokens":433,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T20:32:05.744604+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On the same CoderEval/DevEval splits, replace SADGS with random retention of the same token budget (or force-drop contexts that share the draft’s external APIs) and check whether Pass@1 and the reported time/token gains over RAG and LongCodeZip disappear.","supporting_citations":[],"review_version":1}