Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

When a coding agent must edit code, the signal lives in the edited code itself—summaries and surrounding structure add almost nothing.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 17:39 UTC pith:FJUAZMFW

load-bearing objection A carefully instrumented null: under oracle localization, the act signal is basically the edited code itself, and structured remainder context adds nothing over dropping it. the 2 major comments →

arxiv 2607.09691 v1 pith:FJUAZMFW submitted 2026-06-19 cs.LG cs.AI

What Context Does a Coding Agent Actually Need to Act?

Brian Sam-Bodden This is my paper
classification cs.LG cs.AI
keywords coding agentscontext compressionSWE-benchcode representationact-contextoracle localizationissue resolutionfind/act boundary
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks what context a coding agent actually needs at the moment it edits code, not when it searches for where to edit. Holding localization fixed with an oracle and varying only how code is represented, it scores real issue resolution on every multi-file instance of SWE-bench Verified under a protocol frozen before any data. The answer is minimal: natural-language summaries answer almost none of the behavioral questions that source answers, and a frontier model’s summaries fail as badly as a small model’s—so the gap is the representation class, not summarizer quality. Rendering a file’s non-locus remainder as UML skeletons and signatures resolves no more issues than deleting that remainder outright. Compressed context matches whole files at roughly one-third the tokens. The same instrument also measures a noise floor: temperature-0 API inference flips about nine percent of per-instance outcomes between byte-identical runs.

Core claim

With localization fixed by oracle on every multi-file SWE-bench Verified instance, the actionable signal for editing lives in the code being edited itself. Natural-language summaries answer 4 of 45 test-minted behavioral probes that source answers 27 of 45 times—identically for frontier and 3B summarizers—and no summary succeeds where source fails. Rendering a file’s remainder as UML skeletons and signatures resolves no more issues than deleting that remainder (N=70, exact McNemar p=0.75); the registered hypothesis that structured tiers beat keep/drop failed. Compressed context matches whole files at about one-third the tokens: roughly 19K context tokens per resolve versus 94K.

What carries the argument

The resolve@cost instrument: oracle-localized representation arms with per-instance expressibility checks (every gold edit present in every arm’s context), gold-validated Docker environments, deterministic SEARCH/REPLACE-to-diff patch construction, and pre-registered hypotheses scored by the official harness—so only representation varies and every number traces to a raw artifact.

Load-bearing premise

The experiment treats a single-shot, no-execution regime—one completion plus apply-failure retries, with no test feedback or multi-turn repair—as a fair setting in which to measure what representation an agent needs to act.

What would settle it

Run the same oracle-localized N=70 multi-file Verified pool with a multi-turn agent that can execute tests and re-read: if UML-skeleton surrounding context then resolves substantially more issues than keep/drop under the same expressibility gate, the null that surrounding structure carries no act signal is overturned.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A resolved issue can cost about 19K context tokens rather than 94K when localization is fixed and only the edit locus is kept in full.
  • Filling the context window with whole files or natural-language summaries is largely wasted for the act step.
  • Structured digests of surrounding code (skeletons, signatures) add no measurable resolve signal over simply dropping that remainder in the single-shot regime.
  • Temperature-0 API non-determinism of roughly nine percent per-instance flips is a noise floor under small SWE-bench arm gaps.
  • The find/act boundary implies that summary hierarchies are a poor substitute for source when the agent must reason about behavior to edit.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Practical agent stacks may need separate pathways: structure or retrieval for finding the work site, and verbatim edit-locus source for acting.
  • Context budgets should prioritize full function bodies at the edit locus over repository-wide prose or skeleton maps.
  • Whether multi-turn agents with execution feedback can recover signal from structured surrounding context that single-shot agents cannot is an open, testable transfer question.
  • Published single-run SWE-bench deltas smaller than the measured flip rate should be treated as inside noise unless multi-run statistics are reported.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper isolates act-context from localization for coding agents: on every multi-file SWE-bench Verified instance (N=70 after gold-validation exclusion), gold files are held fixed by oracle while only the representation varies (full files vs structured UML/signature tiers vs binary keep/drop). Resolution is scored with the official Docker harness under a protocol frozen before data, with per-instance expressibility gates and deterministic SEARCH/REPLACE o difflib patches. The registered hypothesis that structured tiers beat keep/drop fails (exact McNemar p=0.75). Compressed arms match or nominally exceed full files at roughly one-third the context tokens (~19K vs ~94K tokens per resolve). A separate behavioral-probe study on held-out repos (n=45, test-minted probes, independent-family judge, κ=0.84) finds source answers ~60% of probes vs ~9% for natural-language summaries, with frontier and 3B summarizers scoring identically. The instrument also measures ~9% per-instance flip rate under byte-identical T=0 re-runs.

Significance. If the results hold within the stated regime, they give a clear, falsifiable answer to a practical design question: for single-shot editing under fixed localization, the edited code itself carries the actionable signal, surrounding static structure adds little over deletion, and compressed context can cut tokens by ~3× without losing resolves. The methodological package is a lasting contribution—pre-registered directional hypotheses with published nulls, gold-validated environments, per-instance expressibility proofs, deterministic patch construction, Wilson CIs, exact McNemar, and a measured T=0 noise floor that bounds small SWE-bench effects generally. The summarizer-quality control (frontier vs 3B) cleanly attributes the probe gap to representation class rather than summarizer strength. Together these strengthen the empirical case for minimal, structure-light act-context and supply a reusable resolve@cost instrument.

major comments (2)
  1. [Abstract / §5 / §7] Abstract and §7 state that “the surrounding context hardly matters” and that neither class skeletons nor signatures add anything over dropping the remainder. §5 scopes the null carefully to the thin end of structured artifacts (static name/signature digests) and notes that interaction-level and distance-graded designs were specified but not fully implemented; H3 (call-flow slice) is only directionally positive and non-significant (p=0.375). The abstract should match that scope so the central null is not read as covering all structured surrounding context.
  2. [Abstract / Table 3 / §3.2 / §6] The cost and “matches whole files” claims rest on oracle ceilings with an acknowledged construction asymmetry: compressed arms encode gold edit units at full detail while oracle_full does not (§3.2, §6). Tier-vs-keep/drop is clean; compressed-vs-full mixes compression with location hinting, and the rate comparison is only marginal (keep/drop vs full, McNemar p=0.07). Tokens-per-resolve remains informative, but the abstract’s “matches whole files at a third of the tokens” should explicitly flag the oracle/location-hint caveat so readers do not treat 19K vs 94K as a deployable retrieval result.
minor comments (5)
  1. [Figure 1] Figure 1 caption and body should state more prominently that the answerer is fixed (Sonnet 4.6) across arms and that frontier/3B labels refer only to the summarizer; the current wording is correct but easy to misread on a first pass.
  2. [Table 1 / §3] Table 1 and §3.1 list representation rungs including method-only and name; the resolve@cost arms actually compared are full / tier / keep_drop. A one-line mapping from rung set to the three resolve arms would reduce ambiguity.
  3. [§1 / §5] The ~9% T=0 flip finding is important for the field; consider elevating a short pointer in the introduction or contributions list so it is not buried only in §5 and the abstract.
  4. [Appendix D / §6] Appendix D exploratory results are appropriately quarantined; a single sentence in §6 or the conclusion restating that no main-text claim relies on them would help readers who skip the appendix.
  5. [Table 3 / §6] Token accounting: headline counts use cl100k_base while Anthropic counts are logged for initial prompts only. A brief note that ratios are robust across the two counters (already partly in §6) could sit next to Table 3 for readers comparing absolute costs.

Circularity Check

0 steps flagged

No significant circularity: external SWE-bench harness, test-minted probes with independent judge, pre-registered nulls, and oracle design isolate representation without self-definition or fitted-as-prediction steps.

full rationale

The paper is an empirical measurement study, not a derivation of a first-principles formula. Resolve rates are scored by the official external SWE-bench Docker harness (FAIL_TO_PASS ∧ PASS_TO_PASS) after gold-patch environment validation, not by self-grading. Behavioral probes are minted from held-out test files (never the scored source), answered by a fixed model, and graded by an independent-family judge with reported κ. The summarizer-quality control uses both frontier and 3B writers under identical budgets. Oracle localization and the expressibility gate are disclosed design choices that isolate the representation variable; they do not redefine success in terms of the arms. Hypotheses (H2 tier > keep/drop; H3 slice > keep/drop) were frozen and committed before data, and the nulls are published. No parameter is fitted to a subset and then re-labeled a prediction; no uniqueness theorem or ansatz is imported via self-citation; no known result is merely renamed. The ~9% T=0 flip rate is measured on byte-identical re-runs and used as a noise floor, not as a fitted claim. The work is therefore self-contained against external benchmarks and free of the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

This is an empirical systems paper, not a derivation paper. Load-bearing content is experimental design choices and domain assumptions about SWE-bench as a proxy for act-context need, not free parameters fitted to produce a target constant. No new physical entities are postulated. The main ‘axioms’ are methodological: oracle localization isolates representation; single-shot resolve is informative; the chosen structural rungs represent the structured-artifact family they claim to test.

free parameters (3)
  • summary token budget (~165–180 tokens)
    Fixed budget for NL summary arms in the probe study; controls compression level and is chosen by design, not fitted to maximize the source–summary gap.
  • agent decoding (max_tokens=8000, T=0, ≤2 apply-failure retries)
    Hand-chosen decoding and retry policy shared across resolve arms; affects absolute resolve rates but is held constant across representation comparisons.
  • representation rung set (full-body, method-only, UML-skeleton, signature+doc, name, exclude)
    Discrete structural tiers chosen by the authors; the registered H2 tests only the implemented thin static digests, not interaction-level or distance-graded artifacts they note were never built.
axioms (4)
  • domain assumption SWE-bench Verified multi-file resolve under the official Docker harness is a valid proxy for whether a context representation suffices for an agent to act.
    Central scoring criterion throughout §3–§5; standard in the subfield but still an assumption about ecological validity.
  • domain assumption Oracle localization (gold-changed files shared across arms) cleanly isolates representation effects from retrieval/find effects.
    Stated design premise of the resolve@cost rig (§3.2); compressed arms still encode gold units, which the paper discloses as an asymmetry for compressed-vs-full but not for tier-vs-keep/drop.
  • domain assumption Test-minted behavioral probes graded by an independent-family judge measure actionable act-context signal rather than surface lexical match.
    Underpins the find/act boundary claim in §4; supported by held-out repos and κ=0.84 but still a construct-validity assumption.
  • ad hoc to paper Deterministic SEARCH/REPLACE → difflib unified diffs remove LLM-authored-diff confounds without changing the scientific comparison.
    Instrument choice in §3; development note that model-authored diffs failed often motivates it; comparison remains fair if applied equally.
invented entities (2)
  • resolve@cost instrument (oracle-localized representation arms + expressibility gate + gold-validated envs + deterministic patch builder) independent evidence
    purpose: Measure which code representations suffice for issue resolution at fixed localization with audited cost.
    New experimental apparatus introduced by the paper; independent evidence is the released code and per-instance artifacts, not an external physical prediction.
  • Structural representation rungs (UML-class-skeleton, method-only, signature+doc, etc.) as act-context tiers no independent evidence
    purpose: Operationalize ‘structured surrounding context’ for the registered tier-vs-keep/drop test.
    Defined via deterministic AST extraction; the null applies only to these thin static digests as implemented.

pith-pipeline@v1.1.0-grok45 · 14618 in / 3748 out tokens · 38942 ms · 2026-07-14T17:39:20.047388+00:00 · methodology

0 comments
read the original abstract

A modern coding agent can hold an entire repository in its context window. Most of its reading is wasted -- and the interesting question is not how much context an agent can use, but what it actually \emph{needs}. We study that question at the moment it matters most: when the agent must \emph{edit} code. Separating \emph{finding} the work site from \emph{acting} on it, we hold localization fixed with an oracle, vary only how the code is represented, and score context against real issue resolution on SWE-bench Verified. The answer is starkly minimal. The signal lives in the code being edited itself: natural-language summaries of it answer almost none of the behavioral questions that the source answers ($4/45$ vs.\ $27/45$, held-out repositories, independent judge), and the gap belongs to the representation, not the summarizer -- a frontier model's summaries score exactly as poorly as a 3B model's. The surrounding context hardly matters either: across every multi-file instance in Verified, under a protocol frozen before any data, rendering a file's remainder as UML skeletons and signatures resolves no more issues than deleting that remainder outright ($N{=}70$, exact McNemar $p{=}0.75$). That was our registered hypothesis, and it failed. Compressed context, meanwhile, matches whole files at a third of the tokens: a resolved issue costs $19$K context tokens, not $94$K. The instrument also yielded a finding the field should keep: temperature-0 API inference flips ${\sim}9\%$ of per-instance outcomes between byte-identical runs. That is a noise floor under every small effect reported on this benchmark, including ours. We release the instrument -- gold-validated environments, per-instance proof that every reference edit is expressible from every arm's context, deterministic patch construction, and pre-registered hypotheses whose nulls we publish.

Figures

Figures reproduced from arXiv: 2607.09691 by Brian Sam-Bodden.

Figure 1
Figure 1. Figure 1: The find/act boundary. Behavioral-probe accuracy by context representation (n=45: 15 held-out classes from seaborn/pylint/pytest × 3 probes minted from each class’s test file; the same model — Claude Sonnet 4.6 — reads the context and answers in every arm, so only the representation varies; answers graded by an independent-family judge, κ=0.84 against a second judge). The frontier/3B labels indicate the su… view at source ↗
Figure 2
Figure 2. Figure 2: The rig, end to end. Gold-changed files (oracle localization) are rendered under three arms that vary representation; the agent emits SEARCH/REPLACE blocks; a deterministic builder constructs the diff; the official Docker harness reports resolution [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Where the sample comes from. 500 Verified instances → 71 multi-file (the registered criterion) → 70 after one gold-validation exclusion. The whole pool runs; nothing is sampled. full files structured tiers keep/drop 0.0 0.1 0.2 0.3 0.4 0.5 0.6 resolve rate (Wilson 95% CI) 19/70 23/70 25/70 Resolve, N=70 (pre-registered) full files structured tiers keep/drop 0 20 40 60 80 context tokens per resolve (thousan… view at source ↗
Figure 4
Figure 4. Figure 4: The powered result (N=70). Left: resolve rate per arm with Wilson 95% CIs — the registered tier-vs-keep/drop comparison is null (exact McNemar p=0.754). Right: context tokens per resolved instance: compressed context buys a resolve for 19–26K tokens; whole files pay 94K. This says summaries cannot act; it does not establish that summaries help navigate — in our exploratory navigation test (Appendix D), sum… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CodeNib: A Multi-View Data System for Serving Repository Context to Coding Agents

    cs.SE 2026-07 conditional novelty 6.0

    A per-commit multi-view repository-index system with incremental updates that match independent rebuilds at 8.7x/25.4x median speedups, and context policies that cut agent trajectory tokens 50-87% while holding locali...

Reference graph

Works this paper leans on

13 extracted references · 9 linked inside Pith · cited by 1 Pith paper

  1. [1]

    LocAgent: Graph-guided LLM agents for code localization

    Zhaoling Chen, Xiangru Tang, Gangda Deng, Fang Wu, Jialong Wu, Zhiwei Jiang, Viktor Prasanna, Arman Cohan, and Xingyao Wang. LocAgent: Graph-guided LLM agents for code localization. InAnnual Meeting of the Association for Computational Linguistics (ACL), 2025. arXiv:2503.09089

    Pith/arXiv arXiv 2025

  2. [2]

    From local to global: A graph RAG approach to query-focused summarization.arXiv preprint arXiv:2404.16130, 2024

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. From local to global: A graph RAG approach to query-focused summarization.arXiv preprint arXiv:2404.16130, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Barr, and Sergey Mechtaev

    Haoxiang Jia, Earl T. Barr, and Sergey Mechtaev. Compressing code context for LLM-based issue resolution.arXiv preprint arXiv:2603.28119, 2026

    arXiv 2026

  4. [4]

    LLMLingua: Com- pressing prompts for accelerated inference of large language models.Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LLMLingua: Com- pressing prompts for accelerated inference of large language models.Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. arXiv:2310.05736

    Pith/arXiv arXiv 2023

  5. [5]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR), 2024. arXiv:2310.06770

    Pith/arXiv arXiv 2024

  6. [6]

    Sufficient context: A new lens on retrieval augmented generation systems

    Hailey Joren, Jianyi Zhang, Chun-Sung Ferng, Da-Cheng Juan, Ankur Taly, and Cyrus Rashtchian. Sufficient context: A new lens on retrieval augmented generation systems. In International Conference on Learning Representations (ICLR), 2025. arXiv:2411.06037

    Pith/arXiv arXiv 2025

  7. [7]

    Barr, Federica Sarro, Zhaoyang Chu, and He Ye

    Han Li, Letian Zhu, Bohan Zhang, Rili Feng, Jiaming Wang, Yue Pan, Earl T. Barr, Federica Sarro, Zhaoyang Chu, and He Ye. ContextBench: A benchmark for context retrieval in coding agents.arXiv preprint arXiv:2602.05892, 2026

    arXiv 2026

  8. [8]

    Alibaba LingmaAgent: Improving automated issue resolution via comprehensive repository exploration

    Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. Alibaba LingmaAgent: Improving automated issue resolution via comprehensive repository exploration. arXiv preprint arXiv:2406.01422, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    RepoGraph: Enhancing AI software engineering with repository-level code graph

    Siru Ouyang, Wenhao Yu, Kaixin Ma, Zilin Xiao, Zhihan Zhang, Mengzhao Jia, Jiawei Han, Hongming Zhang, and Dong Yu. RepoGraph: Enhancing AI software engineering with repository-level code graph. InInternational Conference on Learning Representations (ICLR),

  10. [10]

    CodeCompass: Navigating the navigation paradox in agentic code intelli- gence.arXiv preprint arXiv:2602.20048, 2026

    Tarakanath Paipuru. CodeCompass: Navigating the navigation paradox in agentic code intelli- gence.arXiv preprint arXiv:2602.20048, 2026

    arXiv 2026

  11. [11]

    Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. Manning. RAPTOR: Recursive abstractive processing for tree-organized retrieval. InInterna- tional Conference on Learning Representations (ICLR), 2024. arXiv:2401.18059

    Pith/arXiv arXiv 2024

  12. [12]

    SWE-Pruner: Self-adaptive context pruning for coding agents

    Yuhang Wang, Yuling Shi, Mo Yang, Rongrui Zhang, Shilin He, Heng Lian, Yuting Chen, Siyu Ye, Kai Cai, and Xiaodong Gu. SWE-Pruner: Self-adaptive context pruning for coding agents. arXiv preprint arXiv:2601.16746, 2026

    Pith/arXiv arXiv 2026

  13. [13]

    answerable only from this source

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying LLM-based software engineering agents. InProceedings of the ACM International Conference on the Foundations of Software Engineering (FSE), 2025. arXiv:2407.01489. A Agent prompt and decoding The agent receives, for every arm, the byte-identical instruction below; only...

    Pith/arXiv arXiv 2025