{"id":"e3a3d03c-47d7-4666-b912-f09c5955c10c","arxiv_id":"2608.07147","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"DiDPO splits code diffs into similar sub-diff anchors across rollouts and uses those groups to compute fine-grained token-level advantages, outperforming GRPO and GiGPO on coding benchmarks.","lead":"This paper introduces DiDPO, a reinforcement learning method that gives individual code changes inside multi-turn coding trajectories their own credit by grouping similar sub-diffs across different attempted solutions. It reports consistent gains over group-based RL baselines on eight coding benchmarks with about 2.3% extra training time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DiDPO's gain hinges on treating lexically similar sub-diffs as exchangeable causal units; Eq. 12 and Theorem 4.2 do not rule out contextual confounding, so the 4.2-point margin over GiGPO could be a grouping artifact.","rationale":"I read the paper in good faith. DiDPO addresses a real problem: coding-agent actions contain multiple functional changes, and step- or episode-level advantages cannot separate them. The empirical setup is unusually well controlled: all RL baselines start from the same SFT checkpoint, use the same reward, environment, and hyperparameter settings, and the ablations in Table 3 isolate the contribution of AD. The greedy submodular anchor selection in Eq. 10 is a legitimate approximation with a known guarantee, and the reported overhead of 2.3% is plausible given the quadratic but thresholded similarity computation. I therefore do not view the paper as sloppy or overstated in tone; the concern is substantive, not stylistic. The most load-bearing weakness is the identification assumption behind Eq. 12 and Theorem 4.2. The theorem assumes the step return decomposes into a sub-diff-specific causal term plus zero-mean non-causal noise, but the estimator groups sub-diffs purely on text/embedding similarity and compares raw step returns. In a multi-turn coding process, the same edit inserted in different code states, or at different time steps in the trajectory, can have different causal effects. If those effects are correlated with other parts of the rollout, the group average is a biased baseline and AD is not local credit. The reader's weakest assumption identified exactly this exchangeability problem, and I agree with that diagnosis. My concrete test would settle it by going beyond re-deriving the math: a synthetic MDP with known causal structure directly measures whether AD is nonzero for matched-but-irrelevant sub-diffs. If that test passes, the concern is resolved and the paper's core mechanism is credible; if it fails, the mechanism needs revision even though the empirical tables might still hold. The reader also noted missing error bars, theory-implementation mismatch, and opaque filtering; those remain, but the exchangeability assumption is the one that, if false, would strip the 4.2-point average advantage of its stated explanation. Hence the verdict should remain CONDITIONAL rather than flip to REJECT or ACCEPT: the concern is central and currently untested, but it is addressable by the proposed experiment.","tokens_in":19545,"tokens_out":6231,"duration_ms":63172,"concrete_test":"Construct a synthetic multi-edit coding MDP with known causal structure: two code regions, one whose content determines the test reward and one distractor region that has zero causal effect but contains sub-diffs that are textually/embedding-similar across rollouts. Run DiDPO's anchor selection, group formation, and AD computation on rollouts from this MDP. Check whether AD is systematically nonzero for distractor sub-diffs, and whether a policy optimized with DiDPO's objective increases probability of distractor edits; if either holds, the exchangeability assumption in Eq. 8/12 fails and Theorem 4.2's zero-mean condition is violated. Repeating with groups constrained to identical step indices would also isolate the temporal confound in Eq. 12.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism is the diff-level advantage AD in Eq. 12, formed by grouping sub-diffs matched on lexical/embedding similarity in Eq. 8. Theorem 4.2 justifies this by assuming R_i = r(s_i) + xi_i with xi_i zero-mean non-causal noise. That assumption is load-bearing and not guaranteed: a sub-diff's causal contribution depends on the rest of the code state, the order of edits, and whether later steps compensate for it. Two similar-looking sub-diffs in different contexts can have opposite effects, so the group average in Eq. 12 is not an estimate of r(s_i); AD can then reward tokens that merely co-occur with high-return rollouts. A second, independent confound is temporal: R_t is a step-level discounted return, and groups can contain matched sub-diffs from different steps, so AD compares different future horizons, which the theorem's decomposition does not model. AD is the component that Table 3 credits for a 7.5-point APPS margin and Table 1 for a 4.2-point average gain over GiGPO; if this grouping bias is unvalidated, the main empirical advantage is not explained. The paper does not report error bars, which makes it impossible to distinguish a genuine credit-assignment gain from one noisy run of a confounded estimator.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DiDPO, a critic-free policy optimization method for coding agents that decomposes code diffs into sub-diffs, matches similar sub-diffs across rollouts using a groupability score, forms advantage groups, and computes diff-level advantages in addition to trajectory-level advantages. The method is evaluated on eight code-generation benchmarks with two backbones, with ablations and an efficiency analysis, and the authors release an open-source codebase. The central claim is that DiDPO improves coding-agent RL by providing fine-grained, local credit to individual edits without extra rollouts or a learned critic.","tokens_in":19829,"tokens_out":4276,"duration_ms":37835,"significance":"If the empirical gains are reproducible, the paper makes a useful contribution by extracting credit signals from the internal structure of code diffs, an underexplored axis for agentic RL. The comparisons are internally controlled: all RL baselines start from the same SFT checkpoint and share the environment and reward. The paper also provides theoretical bias-variance bounds and releases a codebase, which are concrete assets. However, the central quantitative claims currently rest on tables without error bars and on a theorem whose key decomposition is not validated; the significance is therefore conditional on those points being addressed.","major_comments":[{"comment":"The main tables report point estimates only, while Appendix E states that 'Validation accuracy is averaged over 5 runs' but provides no standard deviation or confidence interval. With evaluation subsets as small as 21 USACO Platinum problems and 53 ICPC Hard problems, differences of one or two points cannot be distinguished from noise; the abstract's 'over 10%' claim and the 4.2-4.9 point margins over GiGPO therefore need error bars or formal statistical tests. This is load-bearing because the paper's primary case is empirical superiority.","section":"Section 5.2, Tables 1 and 2; Appendix E"},{"comment":"The theorem assumes R_i = r(s_i) + xi_i with zero-mean non-causal noise, but R_t in Eq. 12 is a discounted return over future steps, so the term r(s_i) would have to include all future rewards caused by the sub-diff. The assumption that xi_i is zero-mean is not established: later rewards often depend on later actions, not on noise from earlier steps, and a group in Eq. 12 can contain matched sub-diffs from different timesteps with different future horizons. Without a justification or an empirical check, the bias-variance bound does not establish that AD estimates the local causal contribution.","section":"Section 4.5, Theorem 4.2 and Eq. 12"},{"comment":"The method treats lexically and embedding-similar sub-diffs as exchangeable units, but identical-looking edits can have opposite effects depending on the surrounding code state, the order of edits, and whether later steps compensate. The paper does not test this exchangeability assumption. A controlled test, for example a synthetic coding environment where sub-diff effects are known or a leave-one-out style analysis on matched sub-diffs, is needed before the +7.5 APPS-point marginal gain attributed to AD in Table 3 can be interpreted as a credit-assignment signal rather than a grouping artifact.","section":"Section 4.2-4.3, Eq. 8-12"}],"minor_comments":[{"comment":"The abstract states that DiDPO 'exceeds comparable methods by over 10%', but Table 1 shows smaller average margins of 4.2 and 4.9 points over GiGPO; the specific comparison supporting the 10% claim should be identified.","section":"Abstract"},{"comment":"The plot lacks axis labels and the '~2.3%' annotation is not defined; the paper should describe what is being measured and provide labeled axes so the overhead claim can be verified.","section":"Figure 5 (right)"},{"comment":"The filtering of PRIME training data is described only as 'filter out those that belong to the validation set or do not conform to long-horizon code generation patterns'; the concrete filtering rules or a quantitative description of their effect should be reported.","section":"Section 5.1"},{"comment":"The claim that the groupability score 'optimally balances' semantic scope and group mass is an assertion; s0, g0, and K are free hyperparameters, so the selection or tuning procedure for these values should be described.","section":"Section 4.2 and Eq. 9"},{"comment":"The notation a(i)_l for response tokens and a(i)_{t,m} for sub-diff actions should be explicitly mapped, since Eq. 13 combines them without a formal definition of how a response token span maps to a sub-diff action.","section":"Section 4.4 and Eq. 11-13"},{"comment":"The ablation labels are clear, but the claim that the components' combination 'exceeds additive contribution' is not quantified; the paper should report the sum of individual gains alongside the full-model result.","section":"Section 5.3, Table 3 (right)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript contains several self-citations (e.g., refs. 34, 46-48) and relies heavily on unpublished 2026 preprints, which is not a basis for rejection but may warrant a check on related-work coverage and novelty disclosure. The main unresolved issue is whether the diff-level advantage truly isolates causal credit: the theoretical decomposition in Theorem 4.2 is unvalidated and the empirical tables lack uncertainty estimates. Both concerns are addressable within the manuscript's scope, so revision rather than rejection seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: DiDPO does something genuinely new—it treats code diffs as divisible credit units, groups similar sub-diffs across rollouts, and uses that to localize advantage without a critic or extra rollouts. The experiments are internally controlled, and the gains over GRPO and GiGPO are consistent enough across benchmarks that I believe something real is happening. But the theoretical story is much cleaner than the estimator it justifies, and the missing error bars make the headline margins hard to trust.\n\nWhat's solid: The core idea—sub-diff anchors selected by a groupability score that balances semantic scope and group mass—is not in the cited prior work. The ablation table is useful: removing AD drops APPS from 31.3 to 23.8, and removing sub-diff decomposition to 25.0, which shows the local credit signal is doing work. The comparison is fair: all RL baselines start from the same SFT checkpoint, same environment, same reward. The overhead analysis (2.3% training time) is a nice practical detail.\n\nWhere it gets soft: Theorem 4.2 relies on R_i = r(s_i) + xi_i with zero-mean xi_i. That's a strong exchangeability assumption. A sub-diff that looks like another sub-diff can have opposite causal effects depending on the code state, edit order, or whether later steps compensate. The group average in Eq. 12 is then not an unbiased estimate of r(s_i), so the diff-level advantage can reward tokens that merely co-occur with high returns. There's also a temporal confound: groups can mix sub-diffs from different steps, comparing returns over different horizons. The theorem doesn't model that. This is a real gap, but it may be addressable—the paper could either weaken the theoretical claims or test the assumption directly, e.g., by comparing groupings that condition on surrounding code state.\n\nThe other issues are smaller: no error bars despite the appendix saying validation accuracy is averaged over 5 runs; the training-data filtering is described in one sentence; the open-source artifacts aren't verifiable from the preprint. These are fixable.\n\nWho it's for: people working on RLVR for code agents will want to read this. It's a subfield-level contribution, not a field reorganizer.\n\nRecommendation: send it to review, but expect heavy revision before acceptance. The method is worth referee time; the theory and reporting need to catch up.","headline":"DiDPO has a genuinely new idea for local credit assignment in coding-agent RL, and the internally controlled experiments show plausible gains, but the theory overclaims and missing error bars make the headline margins hard to trust.","tokens_in":20368,"tokens_out":2994,"would_cite":true,"duration_ms":26468,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DiDPO claims that RL credit for coding agents should be assigned at the level of functional sub-diffs inside code edits, and reports gains of 4.2–4.9 points over state-grouping baselines.","keywords":["diff-in-diff policy optimization","coding agents","credit assignment","reinforcement learning with verifiable rewards","code diffs","group-relative advantage","submodular anchor selection","long-horizon code generation"],"falsifier":"On tasks with pairs of rollouts sharing a textually identical sub-diff but differing in whether that edit actually determines test outcomes, log DiDPO's assigned diff-level advantages: if the identical sub-diff receives the same advantage despite opposite causal roles, the exchangeability assumption fails; ablating the edit in each context would quantify the bias.","tokens_in":19332,"feed_emoji":"🧩","tokens_out":9684,"duration_ms":83475,"temperature":0.7,"pith_summary":"DiDPO claims that the right unit of credit for training coding agents is not the whole trajectory, the step, or the environment state, but the functional sub-diff inside a code change. It builds those units by matching similar sub-diffs across sampled rollouts, scoring candidate anchors with a groupability score that balances how much code an anchor represents with how many rollouts support it, and then averaging group-relative returns to get a diff-level advantage. This advantage is added to the trajectory-level advantage and projected onto response tokens, with no critic and no extra rollouts. If the claim is right, RL training for coding agents can localize feedback to individual edits, which is what the experiments report: on Qwen2.5-Coder-7B, DiDPO averages 48.4% across eight coding benchmarks, 4.2 points above GiGPO and more than 10% relative above comparable methods, narrowing the gap to much larger models.","feed_headline":"Edit-level RL credit beats state-grouping baselines by 4.2–4.9 points","feed_subtitle":"Grouping similar sub-diffs across rollouts lifts both 7B and 4B coding agents on long-horizon benchmarks.","key_machinery":"The load-bearing mechanism is the pairing of a groupability score with dynamic sub-diff anchors. The score is $GS(c) = (1-e^{-\\bar L(c)})(1-e^{-(n(c)-1)})$, where $\\bar L(c)$ is the average normalized size of the anchor's occurrences and $n(c)$ is the number of supporting occurrences; the saturating factors keep semantic scope and group mass from dominating each other. Anchor selection maximizes the total covered score over sub-diffs, a cardinality-constrained submodular maximization solved greedily with the classical $(1-1/e)$ guarantee. Selected anchors induce advantage groups over aligned sub-diffs, and the diff-level advantage is the normalized difference between a sub-diff's step return and the group mean, combined through $\\hat A_{i,l} = A_E(\\tau^{(i)}) + \\lambda A_D(a_l^{(i)})$ and trained with the clipped objective of Eq. (14). The theoretical part treats sub-diff alignment as a metric correspondence with distortion error $\\epsilon$, proving a local-contrast bias of $O(L\\epsilon)$ and an MSE bound of $O(L^2\\epsilon^2) + O(\\sigma_\\xi^2/m)$, with an $O(\\sigma_\\xi^2)$ contamination term for episode-level broadcasting.","core_discovery":"On the paper's own terms, the discovery is that code diffs produced by coding agents are divisible credit units: a single action at one step contains several functionally distinct sub-diffs, and these sub-diffs recur across rollouts even when whole diffs do not. DiDPO formalizes recurrence by aligning sub-diffs through a similarity threshold, selecting anchors through a groupability score, and grouping occurrences of the same anchor into advantage groups. It then defines a diff-level advantage by normalizing the step return against the average return within the group, and combines it with the trajectory-level advantage in a clipped PPO objective. The paper's central empirical assertion is that this edit-level credit improves long-horizon coding performance on both a 7B and a 4B backbone, with the largest gains on competition-style and interview-style tasks, at roughly 2.3% added training cost.","pith_inferences":["Editorial inference: the same anchor-and-group machinery should transfer to other edit-heavy agent domains, such as notebook cells, document revisions, or configuration patches, where one action contains multiple functional changes; the paper only demonstrates code.","Editorial inference: the theory predicts that the similarity threshold and group mass should co-adapt, since more diverse rollouts make large low-bias groups harder to form; a testable extension would decay the matching threshold over training rather than fixing it at 0.8.","Editorial inference: if sub-diff recurrence really carries credit, then sampling strategies that deliberately generate variant rollouts around the same edit should increase group mass and speed learning, which is a natural but unexplored extension.","Editorial inference: a failure mode not covered by the paper is sparse overlap: on tasks where successful solutions share almost no code text, anchor groups shrink toward size one and DiDPO should degrade toward GRPO; measuring the correlation between group size and gain across tasks would bound when the method helps."],"forward_implications":["On Qwen2.5-Coder-7B, DiDPO raises the eight-benchmark average from 44.2 (GiGPO) to 48.4, and on Qwen3.5-4B from 53.7 to 58.6.","The gain is concentrated on long-horizon tasks: APPS Interview improves by 10.4 points over GiGPO on the 7B backbone, and USACO jumps to 15.6 versus GRPO's 6.8.","The method needs no value critic and no additional environment rollouts; measured training overhead over GRPO is about 2.3 percent.","Ablations make each component load-bearing: removing the episode-level advantage drops APPS from 31.3 to 10.4, removing the diff-level advantage to 23.8, and removing sub-diff decomposition to 25.0.","The saturating groupability score is not incidental: it outperforms an additive score (31.3 vs 24.4 on APPS) and an LLM-based judge (21.5)."],"supporting_citations":[{"why":"State-level grouping baseline that DiDPO extends and outperforms.","marker":"[14]"},{"why":"Supplies the trajectory-level group-relative advantage and PPO objective formulation that DiDPO builds on.","marker":"[39]"},{"why":"Defines the CodeRL+ training pipeline and execution-semantics alignment setting whose data and evaluation protocol DiDPO adopts.","marker":"[24]"},{"why":"Defines the CodeAct thought-action-observation loop and add/delete/none code actions that DiDPO assumes as its interaction model.","marker":"[44]"},{"why":"Justifies greedy anchor selection through the (1-1/e) guarantee for cardinality-constrained submodular maximization.","marker":"[36]"},{"why":"Provides the prime training task source used to build the SFT and RL training sets.","marker":"[12]"},{"why":"Provides the APPS benchmark used to motivate the need for sub-diff grouping and to measure long-horizon coding gains.","marker":"[18]"}],"fun_headline_variants":["DiDPO: Credit assignment via code sub-diffs","Sub-diff grouping lifts coding RL by 10%","Diff-in-Diff Policy Optimization beats baselines","Edit-level rewards improve coding agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that sub-diffs that look similar across rollouts are the same kind of causal change, and that irrelevant trajectory noise averages out within each group; if the same edit helps in one context and hurts in another, the local advantage is biased.","fun_headline_variants_meta":{"raw":{"variants":["DiDPO: Credit assignment via code sub-diffs","Sub-diff grouping lifts coding RL by 10%","Diff-in-Diff Policy Optimization beats baselines","Edit-level rewards improve coding agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000333,"raw_usage":{"total_tokens":1897,"prompt_tokens":1037,"completion_tokens":860,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":800}},"tokens_in":653,"tokens_out":860,"duration_ms":7996,"temperature":1.0,"reasoning_tokens":800,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:56:42.727953+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On tasks with pairs of rollouts sharing a textually identical sub-diff but differing in whether that edit actually determines test outcomes, log DiDPO's assigned diff-level advantages: if the identical sub-diff receives the same advantage despite opposite causal roles, the exchangeability assumption fails; ablating the edit in each context would quantify the bias.","supporting_citations":[{"cited_title":"Group-in-group policy optimization for llm agent training","cited_arxiv_id":null,"evidence_quote":"State-level grouping baseline that DiDPO extends and outperforms."},{"cited_title":"Ex- ecutable code actions elicit better llm agents","cited_arxiv_id":null,"evidence_quote":"Defines the CodeAct thought-action-observation loop and add/delete/none code actions that DiDPO assumes as its interaction model."},{"cited_title":"An analysis of approxima- tions for maximizing submodular set functions—i","cited_arxiv_id":null,"evidence_quote":"Justifies greedy anchor selection through the (1-1/e) guarantee for cardinality-constrained submodular maximization."}],"review_version":1}