{"id":"8a391b56-0a25-436f-8cf0-d8df574adff0","arxiv_id":"2611.16816","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"OMA corrects outcome-statistics multiplicity and objective-mass imbalance caused by stepwise materialization of multi-turn LLM agent trajectories, improving GRPO-based training on ALFWorld by 7-9 percentage points.","lead":"This paper identifies a subtle accounting problem in multi-turn LLM agent reinforcement learning: when a single agent trajectory is expanded into multiple rows and tokens for training, the outcome statistics and objective mass can become biased toward longer (often failed) trajectories.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"No significant objection identified","rationale":"The reader's verdict is CONDITIONAL, with the weakest assumption being that the base optimizer's credit assignment is correct. I agree this is a scope condition, but it is not a load-bearing concern because the paper explicitly states OMA does not redesign within-trajectory credit assignment and the results are conditional on the base optimizer being reasonable. The paper does not claim OMA fixes fundamental flaws in GRPO/GiGPO/GSPO; it claims OMA corrects a separate outer-accounting problem. The reader's other concerns (single seed, no code, mixed WebShop results) are valid limitations but are acknowledged by the authors and do not undermine the central claim for ALFWorld. The paper's own limitations section (b173–b179) is thorough. I find no significant objection that would change the verdict. The headline reflects that the paper's argument is internally consistent, the math is sound, and the experimental evidence, while limited in seed count, supports the claimed effect in the primary setting. The concrete test I propose would address the seed-variability concern without requiring a change in verdict.","tokens_in":21473,"tokens_out":644,"duration_ms":7780,"concrete_test":"Re-run the ALFWorld-3B GRPO comparison with 5 independent seeds instead of 1 (main) or 2 (appendix), and report mean and standard deviation of endpoint success and nAUC. If the 7.0–9.4 percentage point improvement is robust across seeds, the central claim is strengthened; if variance is high, the condition-dependent account still holds but the effect size is less certain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper identifies a genuine outer-accounting problem in stepwise multi-turn LLM agent RL training, proposes a clean two-component correction (OMA), and validates it across multiple environments, model scales, and optimizers. The authors are transparent about limitations: single-seed main experiments, two-seed appendix, mixed results on ordinary WebShop, and the condition-dependent nature of the benefit. The mathematical derivation of outcome-statistics multiplicity and objective-mass imbalance is sound, and the experimental design includes component ablations, frozen-batch gradient audits, and cross-optimizer compatibility tests that support the central claim. The weakest assumption identified by the reader—that the base optimizer's within-trajectory credit assignment is correct—is acknowledged by the authors as a deliberate design choice (OMA does not redesign it), and the paper's results are conditional on this assumption being reasonable. This is not a flaw in the argument but a stated scope condition. The paper does not claim universal improvement; it provides a condition-dependent account and tests it in regimes where expansion–outcome coupling varies. No internal inconsistency or unsupported leap was found. The concern about single-seed experiments is partially addressed by the two-seed appendix, and the authors explicitly note this limitation. The central claim is well-supported for the ALFWorld setting, and the broader applicability is appropriately qualified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper identifies an outer-accounting problem in stepwise multi-turn LLM agent RL training, where trajectory expansion into rows and tokens creates outcome-statistics multiplicity and objective-mass imbalance. It proposes Outcome-Mass Alignment (OMA), a two-component method that computes outcome statistics over unique source trajectories and equalizes aggregate objective mass across trajectories. Experiments on ALFWorld and WebShop with GRPO, GiGPO, and GSPO show that OMA improves performance when expansion couples to outcomes, with ALFWorld gains of 7.0-9.4 percentage points, while WebShop shows mixed effects consistent with weak coupling.","tokens_in":21729,"tokens_out":2021,"duration_ms":18200,"significance":"The paper makes a clear and well-motivated contribution by identifying and formalizing a previously overlooked outer-accounting issue in stepwise multi-turn agent RL. The mathematical derivation is sound, the experimental design is thorough (two environments, two model scales, component ablations, cross-optimizer tests, frozen-batch gradient audit), and the authors are transparent about limitations. The central claim is well-supported for ALFWorld, and the condition-dependent account is appropriately qualified. The method is simple, principled, and compatible with existing optimizers, making it practically useful.","major_comments":[{"comment":"The paper's strongest claim is the ALFWorld improvement, but the main experiments use only a single seed (seed 0). The authors acknowledge this limitation in §Limitations and provide two-seed appendix results for ALFWorld-3B and WebShop-1.5B. However, the two-seed results for ALFWorld-3B show that OMA improves endpoint success over Base in both seeds, but the magnitude varies (87.50% vs 85.16%), and the nAUC advantage over MA is small and inconsistent across seeds (OMA nAUC 60.79% vs 58.41%; MA nAUC 57.45% vs 59.08%). The paper's central quantitative claim (7.0-9.4 pp improvement) is based on the single-seed main results. Given that the authors themselves note that these summaries provide 'limited evidence of run-to-run variation rather than population-level estimates', the main-text claim should be explicitly qualified as a single-seed observation, and the two-seed appendix should be referenced in the abstract and conclusion. This does not invalidate the claim but is a load-bearing issue for the strength of the quantitative evidence.","section":"§4.2, Table 1, and §4.7","authors_acknowledge":"Appendix A adds two-seed experiments for ALFWorld-3B and WebShop-1.5B; these sum- maries provide limited evidence of run-to-run varia- tion rather than population-level estimates.","authors_acknowledge_p":0.94},{"comment":"Mass Alignment's invariance to arbitrary repartitioning is explicitly limited to complete-group reductions. The fractional-mass implementation in Eq. (8) handles partial reductions, but the authors state that 'arbitrary independently normalized repartitioning is outside the invariance claim' (§B). This is a reasonable scope condition, but it means that in practice, when trajectories are split across microbatches with separate normalization, the exact 1/K outer mass guarantee may not hold. The paper should discuss how sensitive the method is to this practical issue, e.g., whether the fractional-mass approximation is empirically close to the ideal, or whether there are known failure modes.","section":"§3.3, Eq. (8), and §B","authors_acknowledge":"Equation (8) in the main pa- per gives the fractional-mass implementation used when a trajectory is split across actor microbatches, but arbitrary independently normalized repartition- ing is outside the invariance claim.","authors_acknowledge_p":0.87},{"comment":"The frozen-batch gradient audit is a nice diagnostic, but it uses only five representative batches (one WebShop, four ALFWorld) plus two additional rows. The selection criteria for these batches are not specified. Are they random? Are they chosen to span a range of expansion ratios? The paper should clarify how these batches were selected and whether they are representative of typical training batches. Without this, the audit is illustrative but not a systematic characterization.","section":"§4.4, Table 4","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The abstract states 'improves matched-horizon endpoint success by 7.0-9.4 percentage points' without noting that this is from single-seed experiments. Consider adding a qualifier such as 'in single-seed main experiments' or referencing the appendix.","section":"Abstract","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Figure 1 is very effective. However, the caption could be more self-contained; it currently refers to 'this pair' without specifying that the numbers come from the example in the text. Consider adding the token counts directly in the caption.","section":"§1, Figure 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The covariance identity is clear, but the notation Ei[ni] for the mean of ni over trajectories could be confused with expectation over a distribution. Since the group size K is fixed, this is fine, but a brief note that Ei is the empirical average over the group would help readers unfamiliar with this notation.","section":"§3.1, Eq. (5)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The nAUC definition uses linear interpolation between validation checkpoints. Since validation is every 5 updates, this is reasonable. However, the paper should state whether the nAUC is computed over the full [0,150] range or only at observed points. Eq. (12) in §C.2 clarifies this, but it would be helpful to mention in §4.1 that nAUC uses trapezoidal integration with linear interpolation.","section":"§4.1, §C.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The component ablation table shows that OA alone underperforms Base on ALFWorld (endpoint 61.7% vs 74.2% at 3B). This is an interesting result that deserves a brief explanation in the main text. The paper notes that OA 'removes row-population sensitivity but leaves token-weighted mass', but it would be helpful to explain why this leads to worse performance, perhaps referencing the covariance identity or the gradient audit.","section":"§4.3, Table 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The learning dynamics figure is dense and the interpretation is relegated to the supplement. Consider adding a brief summary of the key takeaway in the main text, e.g., 'MA reduces entropy and shortens episodes earliest, consistent with its early-stage advantage, while OMA maintains higher entropy and achieves stronger late-stage performance.'","section":"§4.5, Figure 6","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The discussion of AsyncWebRL (Bai et al., 2026) is helpful for positioning. However, the paper could also mention other length-aware methods such as Dr. GRPO (Liu et al., 2025) and λ-GRPO (Wang et al., 2025a) in more detail, as they are cited but not discussed in the context of length weighting.","section":"§5, Related Work","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The conclusion states 'results support a condition-dependent effect' and summarizes the findings well. It could be strengthened by explicitly stating the practical recommendation: when expansion–outcome coupling is strong, use OMA; when it is weak or reversed, OMA may not help and could hurt. This is implicit but not stated as a guideline.","section":"§6, Conclusion","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The two-seed results for ALFWorld-3B show that OA has lower nAUC than Base for both seeds (44.86% and 44.64% vs 54.04% and 47.08%). This reinforces the main-text finding that OA alone is harmful. However, the standard deviations for Base nAUC (4.92) and OA nAUC (0.16) are very different; the OA nAUC is almost identical across seeds. This might be a coincidence or could indicate a bug; the authors should comment on whether this is expected.","section":"Appendix A, Table 5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The audit provenance is commendable. However, the 'Inv.' column reports '12/12' for all rows, meaning all 12 invariance checks passed. It would be helpful to list what the 12 checks are (e.g., row duplication, token inflation, both, for each of OA, MA, OMA).","section":"Appendix G, Table 9","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper uses the term 'outer-accounting' which is clear in context. However, the distinction between 'outer-accounting' and 'within-trajectory credit assignment' could be made even sharper in the introduction by explicitly stating that OMA does not address the question of which actions within a trajectory were good or bad, only how much influence each trajectory has overall.","section":"General","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The paper identifies a real and previously overlooked problem: when you materialize multi-turn agent trajectories stepwise (one response row per turn), the expansion creates two distinct outer-accounting biases—outcome-statistics multiplicity and objective-mass imbalance. The formalization is clean, the proposed OMA correction is principled (two independent components that don't touch the base optimizer's credit assignment), and the experimental design is thoughtful: two environments, two model scales, component ablations, cross-optimizer tests, and a frozen-batch gradient audit. The authors are transparent about limitations—single-seed main experiments, two-seed appendix, mixed results on ordinary WebShop, and the condition-dependent nature of the benefit. This is a solid contribution to the subfield of LLM agent training.\n\nThe main soft spot is the reliance on a single seed for the primary experiments. The two-seed appendix provides only limited additional evidence, and the authors acknowledge this. For a method that could have non-trivial variance, this is a real weakness. The mixed/negative results on ordinary WebShop are honestly reported and consistent with the paper's condition-dependent account, but they do limit the generality of the claimed benefits. The WebShop-SOL contrast changes multiple variables (model scale, termination), so it's not a clean causal test—the authors note this. The weakest load-bearing assumption is that the base optimizer's within-trajectory credit assignment is correct; OMA explicitly doesn't redesign it, so the method's value is conditional on that being reasonable. The paper acknowledges this as a scope condition, not a flaw.\n\nWho this is for: researchers working on multi-turn LLM agent RL training, especially those using stepwise materialization with GRPO or similar group-relative methods. The paper deserves a serious referee; the core idea is novel and the execution is careful. I'd recommend acceptance after the authors address the seed variability concern (more seeds) and ideally release code/data for reproducibility. The paper is not a desk reject.","headline":"OMA identifies a genuine outer-accounting problem in stepwise multi-turn LLM agent RL training; the paper is well-executed and honest about its limits, but the single-seed main experiments and mixed WebShop results mean the central claim is well-supported only for ALFWorld.","tokens_in":22367,"tokens_out":519,"duration_ms":6554},"pith_extraction":null,"created_at":"2026-11-13T12:03:51.240107+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":null,"supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}