Software Engineering · arXiv:2610.35992 v1 · submitted 26 Oct 2026 · 4 references
Before the First Edit? Measuring When Coding-Agent Guidance Actually Arrives
The paper says
“When a coding-agent harness promises guidance before the first source edit, what event does it actually intercept? We audit five separate events: policy assignment, request eligibility, trigger, plan delivery, and observed source-content change.”
more of the abstract
A prescribed-action comparison exposes the coverage of three gate rules but cannot tell us how an agent would continue after receiving a plan. We therefore run local agents on public software issues and hash Python files before and after each executed action. In a pilot-guided, 76-arm study using Qwen2.5-Coder 14B, 15 arms change Python source across seven tasks and three repositories. Guidance precedes every observed first change, but every change uses a structured editor and the study misses its prespecified 20-arm coverage target. Post hoc local tests find seven changed arms that pass the tested target and regression selectors; four of those seven use incomplete regression lists. Public PAUSE and SI2CA trajectories add source-text changes bracketed by adjacent reads, but lack synchronized filesystem states and cannot identify exact first mutations. The audit makes the timing claim testable within a stated source scope. These data do not show that planning improves repair correctness or covers shell-first writes in live agent runs.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 28 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A careful and transparent empirical audit of coding-agent guidance timing, but the headline claim that 'guidance precedes first edit' rests on structured-editor changes only and misses its coverage targets. This line is wrong
1 major objection · 8 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 4 references say.
The paper says, at §5.9, §5.10, §5.11; §7 (Conclusion)
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.91):
“Our own ex- periments also expose limits: source changes come from few tasks, all live first changes use a structured editor, and no complete repair outcomes were graded.”
Pith says: Objection 1 of 1 · major
The central timing claim—'guidance precedes every observed first change'—is supported only for first changes made via the structured replace_text tool. The paper explicitly acknowledges that no live first change used a shell command (§5.9: 'the live cohort therefore does not test an actual shell‑write bypass'; §6.2: 'all first changes used a structured editor, so live shell-bypass prevalence remains unmeasured').
more of objection 1
Nevertheless, the abstract and conclusion phrase the result as a general 'guidance before first edit' statement. The paper should restructure the presentation so that the limitation for shell-first mutations appears immediately after any summary claim, perhaps by qualifying 'observed first Python-content change' as 'first observed Python-content change made via replace_text' in every occurrence outside the detailed sections. Without this adjustment, readers may overinterpret the scope of the audit's positive finding.
8 minor objections
The paper says, at Abstract
Pith says: minor objection 1 of 8
The abstract includes the sentence 'These data do not show that planning improves repair correctness or covers shell-first writes in live agent runs.' This is good but could be strengthened by also stating that every observed first change used a structured editor. Currently that fact appears later in the same paragraph; it should be stated earlier and more prominently.
The paper says, at §1, Introduction
Pith says: minor objection 2 of 8
The term 'Pi' is used without definition on first occurrence (line after 'PAUSE...'). Define 'Pi' as 'PAUSE intervention' or spell out the acronym earlier.
show 6 more
The paper says, at §4.2.1, Intervention-boundary model
Pith says: minor objection 3 of 8
The symbol M_t is defined as 'confirmed source-file delta during execution'. In the local studies, M_t is observed at action boundaries; in the trace analysis, it is approximated. Clarify that the trace-level proxy is not a confirmed delta but an inferred change interval.
The paper says, at Table 2
Pith says: minor objection 4 of 8
The table caption says 'Baseline and PAUSE-assigned runs' but the column header 'Solved B → P' is ambiguous. Add a footnote explaining that these are task-level resolved counts from the public releases, not repeated-measure comparisons.
The paper says, at §5.5, Table 6
Pith says: minor objection 5 of 8
The column 'Changed reads' appears only for the discovered-strategy arm; the standard arm has 'Not applicable'. Consider using a different symbol or an explicit note in the caption to avoid confusion.
The paper says, at §6.2, Threats to validity
Pith says: minor objection 6 of 8
The threats are enumerated but the organization could be tighter. Combine the sections on construct and internal validity into a single threat list with subheadings for each source of ambiguity (e.g., command vs. delta, path eligibility, live configuration, trace observability).
The paper says, at Conclusion
Pith says: minor objection 7 of 8
The sentence 'A full EMSE submission still needs outcome-aware evidence, independent human coding and scientific review, and a public reproduction package with rights-cleared inputs' is appropriate but appears only in §6.2. Move a version of this to the conclusion so that the manuscript's own self-assessment of completeness is prominently visible.
The paper says, at Throughout
Pith says: minor objection 8 of 8
The LaTeX rendering of 'official' appears in multiple places (e.g., 'SWE-bench official', 'official grading'). These should be 'official'. Also check for other encoding artifacts such as missing spaces before citations.
What would settle it
Run 10–20 arms of the 14B local agent setup on a subset of tasks, but programmatically prevent the agent's first eligible action from being a structured editor (e.g., inject a dummy replace_text error so the model falls back to a shell command like python3 -c or cp). Then measure whether guidance (plan) is delivered before the first observed .py content hash change. If every such arm shows guidance before the shell‑first delta, the concern is largely resolved; if some show a change before the plan, the scope limitation becomes a genuine counterexample.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 28 Oct 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee minor revision: 1 major objection, 8 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This paper does the field a service by asking a concrete question: when a coding-agent harness promises guidance before the first source edit, what event does it actually intercept? The five-event model (policy assignment, eligibility, trigger, delivery, source delta) is a useful decomposition, and the multi-turn hash-based audit is a genuinely novel way to test the claim.
What the paper does well: it defines its terms precisely, runs controlled local experiments and public trace analysis, and reports what it finds with unusual candor. The limitations are stated upfront: source changes come from few tasks, all live first changes used a structured editor, and no repair outcomes were graded. The analysis of PAUSE and SI2CA traces shows where trace analysis alone cannot recover exact first mutation times. The paper is methodologically careful and does not oversell.
Where the soft spots are: the central timing claim — guidance precedes every observed first change — is only validated for the 38 first changes that used the structured replace_text tool. Shell-first mutations are not tested in the live runs. The paper acknowledges this explicitly in section 5.9 ("the live cohort therefore does not test an actual shell-write bypass") and section 5.10 ("this live study still does not exercise a shell-write bypass"). The stress test is right: the claim is narrower than a reader might assume. The study also missed its prespecified coverage target (20 changed arms in three repositories). The post hoc local tests use partial regression lists. None of these are fatal — the paper is honest about them — but they mean the headline conclusion is fragile if interpreted broadly.
Who this paper is for: researchers working on coding-agent harness evaluation, especially those interested in intervention fidelity. It is a good reality check on what trace analysis can and cannot establish. It does not show that planning improves repair correctness, and it does not claim to.
Recommendation: this deserves a serious referee. It is careful, transparent, and the limitations are honestly stated. The contribution is methodological — how to audit the timing claim — and the paper would benefit from peer review that pushes for a public artifact and outcome evaluation. I would engage with it.
The strongest objection
The paper's strongest local-cohort finding—'every observed first Python-content change after guidance'—is supported only for the 38 first changes that all used the structured replace_text tool. The paper candidly acknowledges this (§5.9, §5.10, §5.11: 'every first observed delta followed replace_text'; 'the live cohort therefore does not test an actual shell‑write bypass'). However, the same public PAUSE and SI2CA traces that the paper analyses show that shell commands can and do produce source changes (e.g., the sed, python3 -c, cp witnesses in §§5.2, 5.3, 5.5). Because the local agent's gate could have handled a shell‑first mutation differently (the eligibility code in Appendix A shows distinct handling for shell commands under the 'capability' and 'all_requests' policies), the claim that 'guidance precedes first change' has never been checked against the scenario a realistic adversarial or varied agent would most stress: a shell‑first source edit. The gap is explicitly a scope limitation, not an internal contradiction, but it is the single place where the headline conclusion is most fragile because the observational basis is artificially narrow.
The reader's weakest_assumption states that the local studies assume the structured‑editor request is the only path to source mutation. The paper does not assume this; it reports an observed concentration and repeatedly flags the limitation (§5.9, §5.10, §5.11; Appendix C). However, the reader is correct that this concentration makes the central claim less general than it may appear. My concern sharpens the point: the claim has literally zero live‑evidence support for shell‑first mutations, and the paper's own controlled cells (§5.6) show that shell commands can change files without triggering the editor gate. A test that forces a shell‑first edit in live agent runs would directly assess the fragile edge. Since the paper already discloses the limitation and does not claim shell coverage, the verdict should remain CONDITIONAL (UNCHANGED) rather than moving to REJECT; the paper's strength is its honest auditing, not its generality. The concrete test proposed would either confirm the timing claim under the missing condition or expose a genuine gap, moving the verdict to REJECT or strengthening it to ACCEPT depending on outcome.
The referee's report
If taken at face value, the paper provides a rigorous empirical demonstration that the timing of guidance relative to a source edit can be measured and that a gate rule's coverage differs sharply from its engineering promise. The main contribution is methodological: distinguishing five events, using per-action file hashing, and showing conditions under which a delivery claim is falsifiable. The negative finding—that editor-only or command-marker rules leave shell-first mutations untested—is useful for the design of future harnesses and for interpreting existing evaluation benchmarks. However, the limited number of tasks that changed source, the concentration in a few repositories, and the absence of official grading mean the paper does not establish whether planning improves repair outcomes or generalizes across codebases. The significance lies in the audit methodology and the concrete scope limitations it reveals, not in a broad positive conclusion.
Pith's reading of the paper (a machine's summary, not the authors')
This paper audits the claim that a coding-agent harness delivers guidance before the first source edit. The author separates five distinct events — policy assignment, request eligibility, trigger recording, plan delivery, and observed source change — and shows that they do not always align. Through controlled single-action tests, local multi-turn agent runs with per-action file hashing, and reanalysis of public PAUSE and SI2CA traces, the paper finds that every observed first Python-content change in its local studies did follow guidance, but all those changes used a structured editor rather than a shell write. Public traces lack synchronized filesystem snapshots, so they cannot confirm the exact timing of the first mutation. The study also fails its own prespecified coverage target (20 mutation-bearing arms), and none of the local changes were graded for repair correctness. The core insight is that a harness's intervention is only as early as the requests its gate can see and the plans it actually delivers.
What carries the argument. The central machinery is an action-level source-state observer that SHA-256-hashes every regular Python file before and after each executed agent action, combined with a five-event audit model that distinguishes assignment, eligibility, trigger, delivery, and confirmed delta. This allows the author to place the first durable content change relative to the delivered plan. The local harness implements four gate policies (editor-only, editor-plus-literal-markers, any-writable-tool, and all-requests) and calls a Planner at the first eligible request, discarding that request before the agent continues. The controlled 39-action suite and three multi-turn cohorts (7B development, 7B adaptive follow-up, 14B small-patch) are the vehicles for the timing comparison.
Core claim. The paper establishes that the widely made promise 'guidance before the first source edit' is not a single event but a chain of five independent observations: policy assignment, eligibility test, trigger record, plan delivery, and confirmed source delta. Using per-action SHA-256 hashing of Python files in local agent runs, the author shows that every observed first Python change was preceded by a delivered plan — but every such change used the replace_text structured editor. The public PAUSE and SI2CA trajectories record commands and final patches but not per-tool filesystem states, so they cannot place the exact instant of the first mutation. The audit makes the timing claim testable within a stated source scope, and it forces any future claim to specify which gate rule is used and what mutation channels it covers.
The premise it rests on. The local studies assume that the structured-editor request is the only path to source mutation, because all observed first changes used replace_text; however, shell-first writes are not ruled out and the public traces show that such writes can occur and bypass the gate.
If the paper is right (Pith's inferences, not the authors')
Future evaluations of coding-agent harnesses must log ordered tool-call IDs, eligibility decisions, delivery events, and workspace state before and after every action to support timing audits.
Public trajectory datasets (PAUSE, SI2CA) cannot substitute for per-action filesystem snapshots; they provide only a broad proxy for source change and cannot identify the exact first mutation.
Gate rules that only watch structured editor requests miss shell-first writes, which public traces show do occur; a stronger pre-mutation guarantee requires gating every writable tool surface or enforcing a write barrier.
Task-paired arm comparisons (PAUSE baseline vs. treated) are not individual causal estimates because the draws are independent; conditioning on delivery or final patch selects on post-assignment events.
The paper's audit framework makes delivery timing empirically testable and provides a template for reporting intervention fidelity in agent studies.
If extended beyond Python and structured editors, the same per-action hashing approach could reveal how often shell-first writes actually bypass editor-based gates in live agent runs — a quantity the present study could not measure because its local agents never exercised that path.
The paper implies that many existing coding-agent benchmarks may overstate the coverage of planning interventions, since they typically check only final patches and not the sequence of mutations relative to guidance.
The failure to reach the prespecified 20-arm coverage target across three cohorts suggests that achieving a diverse set of mutation-bearing agent runs in an open-source setting is intrinsically difficult, which may limit the generalizability of any single timing claim.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
When a coding-agent harness promises guidance before the first source edit, what event does it actually intercept? We audit five separate events: policy assignment, request eligibility, trigger, plan delivery, and observed source-content change. A prescribed-action comparison exposes the coverage of three gate rules but cannot tell us how an agent would continue after receiving a plan. We therefore run local agents on public software issues and hash Python files before and after each executed action. In a pilot-guided, 76-arm study using Qwen2.5-Coder 14B, 15 arms change Python source across seven tasks and three repositories. Guidance precedes every observed first change, but every change uses a structured editor and the study misses its prespecified 20-arm coverage target. Post hoc local tests find seven changed arms that pass the tested target and regression selectors; four of those seven use incomplete regression lists. Public PAUSE and SI2CA trajectories add source-text changes bracketed by adjacent reads, but lack synchronized filesystem states and cannot identify exact first mutations. The audit makes the timing claim testable within a stated source scope. These data do not show that planning improves repair correctness or covers shell-first writes in live agent runs.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-28
The authors' main claim, as the classifier picked it out 0.81
“These data do not show that planning improves repair correctness or covers shell-first writes in live agent runs.”
How the main claim is stated: asserted 0.98
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.98 · human participants 0.05 · controlled comparison 0.17 · randomized assignment 0.07 · states quantitative result 0.97 · compares with prior results 0.63 · mentions code or data availability 0.06
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| When a coding-agent harness promises guidance before the first source edit, what event does it actually intercept? | gap 0.55 | 0.06 | 0.03 | unclear 0.98 |
| We audit five separate events: policy assignment, request eligibility, trigger, plan delivery, and observed source-content change. | method 0.81 | 0.91 | 0.03 | asserted 0.84 |
| A prescribed-action comparison exposes the coverage of three gate rules but cannot tell us how an agent would continue after receiving a plan. | gap 0.55 | 0.70 | 0.06 | asserted 0.99 |
| We therefore run local agents on public software issues and hash Python files before and after each executed action. | method 0.99 | 0.96 | 0.03 | asserted 0.91 |
| In a pilot-guided, 76-arm study using Qwen2.5-Coder 14B, 15 arms change Python source across seven tasks and three repositories. | method 0.90 | 0.95 | 0.05 | asserted 0.99 |
| Guidance precedes every observed first change, but every change uses a structured editor and the study misses its prespecified 20-arm coverage target. | result 0.93 | 0.95 | 0.29 | asserted 1.00 |
| Post hoc local tests find seven changed arms that pass the tested target and regression selectors; four of those seven use incomplete regression lists. | result 1.00 | 0.95 | 0.09 | asserted 1.00 |
| Public PAUSE and SI2CA trajectories add source-text changes bracketed by adjacent reads, but lack synchronized filesystem states and cannot identify exact first mutations. | limitation 0.47 | 0.14 | 0.05 | asserted 1.00 |
| The audit makes the timing claim testable within a stated source scope. | interpretation 0.52 | 0.88 | 0.14 | asserted 0.88 |
| These data do not show that planning improves repair correctness or covers shell-first writes in live agent runs. | limitation 0.73 | 0.81 | 0.60 | asserted 0.98 |
What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 1.00
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-28: 1 request, 7,416 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (8)
Figures are the authors', reproduced from the paper.
References (4)
- [1] • AgentLens contributors (2026). AgentLens: multi-session coding-agent trajectory harness and shadow-git state tracking . Open-source implemen- tation; pinned revision in the replication guide. • Fan, R.-Z., Zhang, Z., Ma, S., Hu, Y., Wang, S., Song, K., Liu, F., Zamani, H., a... on PithPith's reviewarXiv
- [4] • Oderinwale, H. (2026). Agent trajectories as pro- grams: fingerprinting and programming coding- agent behavior . arXiv:2606.16988. ProcGrep im- plementation. • Patel, V., Tan, B., Xue, J., and Sun, Y. (2026). PAUSE: Search-Free Steering of Cod- ing Agents Before Their First ... on PithPith's reviewarXiv
- [2024] Lite development dataset. • Kapner, B., Soceanu, C., Petrunin, A., and Gart- ner, H. (2026). Scanning the Harness: Configura- tion Exposures in AI Coding-Agent Supply Chains . arXiv:2609.07360v3. • Mirzoian, V. (2026). Retrace. Project documen- tation and implementation, inspe... on PithPith's reviewarXiv
- [2026] • Nian, Y., Yuan, A., Zhang, H., Li, J., Li, L., Hu, X., Wei, H., Xiao, X., Xiao, C., and Zhao, Y. (2026). Auditable Agents. arXiv:2604.05485, version on PithPith's reviewarXiv
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment