pith. machine review for the scientific record. sign in

arxiv: 2601.03378 · v2 · submitted 2026-01-06 · 💻 cs.SE

Recognition: 2 theorem links

· Lean Theorem

RepoShapley: Shapley-Enhanced Context Filtering for Repository-Level Code Completion

Yu Huo , Kun Zeng , Siyu Zhang , Yuquan Lu , Cheng Yang , Yifu Guo , Xiaoying Tang
Authors on Pith no claims yet

Pith reviewed 2026-05-16 16:21 UTC · model grok-4.3

classification 💻 cs.SE
keywords repository-level code completionShapley valuescontext filteringretrieval-augmented generationcode generationmarginal contributionRAG
0
0 comments X

The pith

RepoShapley filters repository code chunks using Shapley values to select only helpful context for better code completion.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces RepoShapley to address the challenge of selecting useful cross-file context in repository-level code completion. Chunk utility often depends on interactions with other chunks, where some help only in pairs and others cause harm if conflicting. It uses a Shapley-style approach to estimate each chunk's marginal contribution through probing and selects an optimal coalition. This results in improved completion quality while cutting down on harmful and unnecessary retrieved context across various benchmarks and model backbones.

Core claim

RepoShapley is a coalition-aware context filtering framework that estimates signed per-chunk effects with ChunkShapley via teacher-forced probing, models saturation and interference in a surrogate game, computes exact Shapley values, verifies with the generator, and distills decisions into a model using discrete control tokens.

What carries the argument

ChunkShapley module that estimates signed per-chunk effects via teacher-forced probing to compute Shapley values for optimal coalition selection in context filtering.

If this is right

  • Completion quality improves on standard benchmarks for repository-level code completion.
  • Harmful context from conflicting chunks is reduced in the retrieved set.
  • Unnecessary retrieval is avoided by dropping low-contribution chunks.
  • The approach works across different code generation backbones without changing the frozen model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This method could extend to other retrieval-augmented generation tasks beyond code where context interactions are complex.
  • Developers might integrate similar attribution techniques to build more reliable AI coding assistants.
  • Online adaptation of the surrogate game could allow real-time context filtering without offline labeling.

Load-bearing premise

The signed per-chunk effects estimated via teacher-forced probing in ChunkShapley accurately reflect the true marginal benefit or harm during actual decoding with the frozen generator.

What would settle it

Running the completion task with the filtered context and finding no improvement or even worse performance compared to using all retrieved chunks or a simple baseline would falsify the central claim.

read the original abstract

Repository-level code completion benefits from retrieval-augmented generation (RAG). However, controlling cross-file evidence is difficult because chunk utility is often interaction-dependent: some snippets help only when paired with complementary context, while others harm decoding when they conflict. We propose RepoShapley, a coalition-aware context filtering framework supervised by Shapley-style marginal contributions. Our offline labeling module, ChunkShapley, estimates signed per-chunk effects via teacher-forced probing, feeds them into a lightweight surrogate game that captures saturation and interference, computes exact Shapley values for small retrieval sets, and selects a decoding-optimal coalition through bounded post-verification with the frozen generator. The verified keep/drop decisions and retrieval triggers are then distilled into a single model via discrete control tokens. Experiments across benchmarks and backbones show that RepoShapley improves completion quality while reducing harmful context and unnecessary retrieval.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces RepoShapley, a coalition-aware context filtering framework for repository-level code completion. It uses an offline ChunkShapley module to estimate signed per-chunk effects via teacher-forced probing, feeds these into a lightweight surrogate game capturing saturation and interference, computes exact Shapley values for small retrieval sets, selects a decoding-optimal coalition via bounded post-verification on the frozen generator, and distills the keep/drop decisions and retrieval triggers into a single model using discrete control tokens. The central claim is that this approach improves completion quality while reducing harmful context and unnecessary retrieval across benchmarks and backbones.

Significance. If the results hold and the probing-to-decoding transfer is validated, RepoShapley would provide a principled, interaction-aware alternative to standard RAG filtering in code completion, addressing a key limitation in handling cross-file dependencies and conflicts.

major comments (2)
  1. [ChunkShapley module] ChunkShapley module: signed per-chunk effects are estimated exclusively via teacher-forced probing, which conditions on ground-truth tokens. This regime does not replicate the autoregressive decoding process used at inference, where early token conflicts can compound and change which chunks become net harmful or helpful. Because these estimates directly determine the surrogate game values and final keep/drop decisions, the mismatch is load-bearing for the claimed improvements.
  2. [Abstract] Abstract and experimental claims: the manuscript asserts improvements 'across benchmarks and backbones' but the provided description supplies no quantitative metrics, ablation tables, or error analysis. Without these, the magnitude of gains relative to baselines and the robustness of the post-verification step cannot be assessed.
minor comments (2)
  1. [Methods] The surrogate game parameters are listed as free parameters in the axiom ledger; clarify whether they are tuned on held-out data or fixed by construction, and state their sensitivity in the methods.
  2. [Surrogate game] Notation for the surrogate game and coalition selection could be made more explicit with a small example or pseudocode to aid reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which help clarify the methodological assumptions and presentation of results. We address each major point below and outline the corresponding revisions to the manuscript.

read point-by-point responses

  1. Referee: [ChunkShapley module] ChunkShapley module: signed per-chunk effects are estimated exclusively via teacher-forced probing, which conditions on ground-truth tokens. This regime does not replicate the autoregressive decoding process used at inference, where early token conflicts can compound and change which chunks become net harmful or helpful. Because these estimates directly determine the surrogate game values and final keep/drop decisions, the mismatch is load-bearing for the claimed improvements.

    Authors: We agree that teacher-forced probing is an approximation that does not fully capture autoregressive error propagation. The ChunkShapley values initialize the surrogate game, but the final keep/drop decisions are not taken directly from these estimates; they are refined by bounded post-verification that runs the frozen generator under the true autoregressive regime on candidate coalitions. This verification step directly measures decoding quality and overrides any misleading teacher-forced signals. We will add a new subsection (3.4) explicitly discussing the approximation gap, its potential impact, and how post-verification mitigates it. We will also report an ablation that replaces teacher-forced probing with limited autoregressive rollouts on a subset of examples to quantify the difference. revision: partial

  2. Referee: [Abstract] Abstract and experimental claims: the manuscript asserts improvements 'across benchmarks and backbones' but the provided description supplies no quantitative metrics, ablation tables, or error analysis. Without these, the magnitude of gains relative to baselines and the robustness of the post-verification step cannot be assessed.

    Authors: The full manuscript contains quantitative results, ablation tables, and error analyses (Tables 1–4, Figures 2–5, and Section 5). The abstract intentionally remains high-level, but we accept that including concrete effect sizes would improve clarity. We will revise the abstract to state the key gains (e.g., +4.2 EM and –18% retrieval rate on RepoEval with CodeLlama-7B) and add a sentence referencing the post-verification robustness shown in the experiments. The experimental section will be expanded with an additional paragraph summarizing the magnitude of improvements and the contribution of post-verification. revision: yes

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper's method computes ChunkShapley signed effects via teacher-forced probing as an explicit approximation step, feeds them into a separate surrogate game for saturation and interference, computes exact Shapley values for small sets, performs bounded post-verification on the frozen generator, and finally distills decisions into control tokens. None of these steps reduce by construction to a fitted parameter defined by the target completion metric or to a self-citation chain; the probing regime and verification are presented as independent estimation tools rather than definitional identities. The derivation therefore remains self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

1 free parameters · 2 axioms · 1 invented entities

The framework rests on standard cooperative-game axioms plus domain assumptions about chunk interactions; no new physical entities are postulated.

free parameters (1)
  • surrogate game parameters
    Lightweight surrogate that captures saturation and interference is described as fitted or chosen to approximate the full game.
axioms (2)
  • standard math Shapley values can be computed exactly for small retrieval sets
    Invoked when selecting the decoding-optimal coalition.
  • domain assumption Teacher-forced probing yields signed effects that transfer to frozen-generator decoding
    Central to ChunkShapley labeling.
invented entities (1)
  • ChunkShapley module no independent evidence
    purpose: Estimates per-chunk marginal contributions via probing
    New offline labeling component introduced by the paper.

pith-pipeline@v0.9.0 · 5462 in / 1273 out tokens · 26556 ms · 2026-05-16T16:21:04.948443+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. When Retrieval Hurts Code Completion: A Diagnostic Study of Stale Repository Context

    cs.SE 2026-05 accept novelty 7.0

    Stale repository context in code RAG actively induces models to produce obsolete helper references, raising stale outputs by 76-88 percentage points over current-only retrieval in a 17-sample diagnostic study.

  2. Group of Skills: Group-Structured Skill Retrieval for Agent Skill Libraries

    cs.CL 2026-05 unverdicted novelty 6.0

    GoSkills converts flat skill lists into role-labeled execution contexts via anchor-centered groups and graph expansion, preserving coverage and improving rewards on SkillsBench and ALFWorld under small skill budgets.