{"id":"4f56c999-84f5-47a3-9d79-49c214f0e28a","arxiv_id":"2607.21419","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adaptively expanding and compressing prompt-scaffold guidance during RL lets LLM agents train to competitive performance without any skill library at test time.","lead":"This paper introduces PATS, a training-time system that adaptively adds and removes text guidance ('scaffolding') during reinforcement learning of LLM agents. It reports that agents trained this way perform on par with stronger skill-based methods while using 25–50% fewer tokens at inference.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Refiner semantic reliability is unmeasured: deterministic checks only verify schema and evidence count, so misleading skill edits could be injected and the claimed benefit of policy-aware adaptation is not isolated from content correctness.","rationale":"The reader's weakest assumption points to the refiner's semantic validity, and this is indeed the most load-bearing concern for the central claim. PATS is fundamentally a system that edits natural-language guidance based on evidence; if those edits are semantically wrong, the scaffold harms rather than helps, and the method's novelty collapses into warmup plus scheduling. The paper provides only syntactic validation and anecdotal audits, so the risk is unresolved. Other concerns (token-claim overstatement, controller threshold tuning, warmup attribution) are real but secondary: the warmup is already controlled by the w/o-training-scaffold ablation, the token claim is explicitly deployment-only in the experiments, and threshold tuning is a standard hyperparameter issue. The refiner concern, by contrast, strikes at the core mechanism. The proposed test—manual annotation of committed edits plus a shuffled-content control—would directly quantify whether the semantic content of the refiner matters. If errors are rare and shuffled content performs similarly, the concern is mitigated; if errors are common or shuffled content degrades performance, the paper needs to add a semantic filter or report quality metrics. Since the reader already reached CONDITIONAL on a related basis and our attack does not move the verdict, UNCHANGED is appropriate.","tokens_in":23643,"tokens_out":10007,"duration_ms":103908,"concrete_test":"On the seed-0 ALFWorld log (801 refiner calls), sample 100 committed edits across EXPAND/REVISE/COMPRESS. Have two annotators judge, against the evidence cards, whether each edit is (a) semantically entailed, (b) actionable, and (c) non-misleading; compute inter-annotator agreement and error rate. In parallel, run one PATS seed with the refiner replaced by a 'shuffle' baseline that outputs schema-valid but content-random principles sampled from the same evidence cards. If the shuffled policy performs within ~2 SR points of real PATS, the semantic content is not load-bearing; if it drops substantially, the refiner's correctness is essential and the paper must report quality metrics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The adaptation loop's output quality is the linchpin of the method, yet the paper never measures the semantic validity of the refiner's proposals. Section 3.3 (Primitive-based Skill Refine) and Appendix B.1 describe a frozen qwen2.5-7b-refiner service whose outputs are checked only for schema validity, duplication, budget, and a minimum evidence count (two supporting cards). Nothing verifies that a proposed principle is entailed by the evidence, is actionable, or is not actively misleading. Because the scaffold is injected into every rollout prompt (Equation 7), a bad edit changes the training distribution for the policy. If the refiner systematically proposes off-target or hallucinated skills, the policy could be trained on spurious context, and the observed gain over GRPO could come from the scaffold-conditioned SFT warmup (+5.7 SR in Table 3) plus the mere presence of dynamic context, not from correct policy-aware content. The paper's own audit (Appendices B.8-B.10) is anecdotal: it shows one correct lineage and one commit, not an error rate. This is the weakest point in the causal chain from 'policy-aware adaptation' to 'competitive scaffold-free performance.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PATS, a policy-aware training scaffold for agentic RL. At each RL iteration, rollout groups are converted into evidence cards; a competence controller chooses among EXPAND, REVISE, COMPRESS, and FORCED_PRUNE based on a task-type success-rate EMA and scaffold pressure; a frozen Qwen2.5-7B refiner proposes schema-constrained edits that are deterministically validated and atomically committed; the resulting scaffold is injected into rollout prompts during training and removed at deployment. The paper claims that PATS achieves performance competitive with SOTA skill-based baselines while using 25%–50% fewer deployment tokens, based on experiments on ALFWorld, WebShop, and seven search-augmented QA benchmarks with Qwen2.5-1.5B/7B, full test sets, three seeds, and component ablations.","tokens_in":23940,"tokens_out":6299,"duration_ms":60437,"significance":"If the empirical claims hold, the paper makes a useful conceptual contribution: reframing skills as a disposable, policy-conditioned training scaffold rather than as persistent test-time artifacts. The strengths are real: evaluations use fixed full test sets; results are reported as three-seed means with standard deviations; ablations isolate components (online adaptation, REVISE, group evidence, interface initialization); the appendices provide detailed schemas, budgets, operation logs, and audit trails; and the code is publicly linked. The paper also states its own limitations clearly, notably that the Stage-0 internalization study is a controlled diagnostic rather than a general skill-acquisition claim. However, two issues currently sit at the load-bearing points of the central claim: the semantic reliability of the refiner's edits is not measured, and the headline comparison to GRPO conflates the scaffold-conditioned warmup with the online adaptation mechanism. Both are fixable with additional analysis, so the appropriate disposition is major revision rather than rejection.","major_comments":[{"comment":"The central causal claim is that policy-aware adaptation improves training by injecting correct, useful guidance. The validator described in §3.3 and Appendix B.1 checks only schema validity, evidence-card count, duplication, budgets, and capacity; it does not verify that a proposed principle is entailed by the evidence, is actionable, or is not misleading. Because the scaffold is inserted into every rollout prompt (Eq. 7), a bad refiner proposal changes the training distribution. The paper's own audit (Appendices B.8–B.10) shows one correct lineage and one commit, and explicitly disclaims causal attribution; it provides no error rate. I request a quantitative audit of refiner outputs across seeds and task types—e.g., human or LLM-judge annotations of semantic validity, usefulness, and harmfulness—plus a report of the fraction of commits that were later reverted or contradicted by subseq","section":"§3.3, Appendix B.1, Appendices B.8–B.10"},{"comment":"The headline comparison to GRPO conflates two effects. In Table 3, 'w/o training scaffold' (same interface-initialized checkpoint, no online scaffold) reaches 73.57±0.89 SR on 1.5B ALFWorld, while the GRPO baseline in Table 1 is 67.86±3.50; thus the scaffold-conditioned SFT warmup alone accounts for +5.7 SR, and the online scaffold contributes the remaining +7.1 of the reported 12.9-point end-to-end gain. Since the warmup is trained on public SkillRL-format data (Appendix B.1), the Table 1 comparison attributes to PATS an improvement that is partly a data/SFT effect. Also, Table 3's 'w/o interface initialization' row (47.86±3.29) shows that without the warmup PATS is far below GRPO. I request that the paper present the decomposition 'GRPO vs. warmup-only vs. warmup + online scaffold' as the primary reading, and that all abstract/conclusion claims specify whether they refer to the full PA","section":"§4.1, §4.2, Tables 1 and 3"},{"comment":"The '25%–50% fewer tokens' claim is stated in the Abstract and Conclusion as if it were a global resource reduction, but the metric in Table 1 is explicitly deployment interaction tokens only. Training-time service costs are substantial: the audited 1.5B run logs 831 refiner requests, 44.1M input characters, and ~180 GPU-hours (Appendix B.1). If the claim is about total compute or total token consumption, the current statement is misleading; if it is about deployment tokens, the Abstract should say 'deployment tokens' and the paper should provide the full training-time token/cost breakdown so readers can assess the end-to-end efficiency trade-off.","section":"Abstract, §4.1, Table 1, Appendix B.1"}],"minor_comments":[{"comment":"The Bernoulli calculation is correct but does not determine the controller thresholds; the appendix itself says the thresholds remain empirical. Please state explicitly in the main text that the intermediate-competence argument is motivational and not a parameter-free derivation, to avoid an unintended implication.","section":"Appendix C and §3.3"},{"comment":"The 'Avg.' row appears to be a macro average across seven datasets of very different sizes (NQ 3,610 examples vs. Bamboogle 125). Please define whether this is macro or micro, and consider reporting the micro average as well, since the conclusion depends on the pooled number.","section":"Table 2"},{"comment":"The figure would be easier to interpret with explicit labels on both y-axes and a statement of whether the PATS curve is evaluated with or without the scaffold at validation. Currently the caption leaves the evaluation condition implicit.","section":"Figure 1"},{"comment":"Minor wording/typo: 'Analyze behavioral shifts' should be 'Analyzing behavioral shifts'; also, the fact that the NLL set covers only two gamefiles should be stated in the main text near the claim, not only in Appendix A.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid empirical systems paper and the scaffolding idea is worth publishing if the two load-bearing gaps are closed: a quantitative audit of refiner semantic quality, and a clear decomposition of warmup vs. online adaptation in all headline claims. The Bernoulli appendix is not a circularity problem. I would not reject the paper; with the requested analyses it could become acceptable. No concerns about citation ethics beyond noting that the warmup's reliance on public SkillRL data should be clearer in the main text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PATS is worth taking seriously. The central idea—treating skills as a disposable, policy-conditioned training scaffold rather than a persistent artifact—is genuinely useful, and the paper backs it with careful ablations and full test-set evaluations. The claim that a policy can internalize scaffold knowledge and be deployed without external skills is supported. The strongest evidence is the ablation table: removing the online scaffold costs 7.1 SR points, removing REVISE costs 12.6, and the scaffold-free model still beats GRPO by a meaningful margin. The offline replay analysis (static skills help inference but narrow reward variation) is a good mechanistic contribution.\n\nThe soft spots are real but not fatal. First, the headline comparison to GRPO conflates the scaffold-interface warmup with online adaptation. Warmup alone accounts for about 5.7 points of the 12.9-point gain on ALFWorld 1.5B; the online scaffold contributes the remaining roughly 7 points. The ablation section addresses this, but the abstract and intro overstate the end-to-end comparison. Second, the abstract's '25%-50% fewer tokens' is deployment-only; training pays a substantial refiner service cost (44M input characters in one audited run). That is fine if stated, but it is not in the abstract. Third—the stress-test point—the semantic reliability of the frozen refiner is unmeasured. Deterministic validation checks schema, evidence count, duplication, and budgets, but not whether a proposed principle is entailed by the evidence or non-misleading. A bad edit injected into every rollout prompt could train the policy on spurious context. The ablations partly mitigate this: REVISE is the largest single component, and 'Frozen after step 50' hurts, so content plausibly matters. But an explicit semantic error rate or a larger manual audit would have strengthened the causal chain.\n\nMinor issues: controller thresholds are tuned per domain without a stated protocol, and the search-augmented QA comparison leans on several baseline numbers reported by SkillRL. These are not disqualifying.\n\nOverall, this is a serious piece of work, honestly reported (the authors flag their own diagnostic limitations). It deserves peer review. Send it out; ask the referee to push for a quantified refiner audit and a more careful framing of the warmup-vs-adaptation decomposition.","headline":"PATS is a solid, well-ablated paper with a real conceptual shift, but headline claims need qualification and the refiner's semantic reliability is an unmeasured dependency.","tokens_in":24443,"tokens_out":3623,"would_cite":true,"duration_ms":36090,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PATS reframes skills as a disposable training scaffold: it expands, revises, and compresses context during RL, then discards it, matching state-of-the-art performance on agentic benchmarks while using 25–50% fewer deployment tokens.","keywords":["reinforcement learning","LLM agents","training scaffold","skill internalization","curriculum learning","GRPO","token efficiency","agentic RL"],"falsifier":"Run PATS with the same controller and budgets but replace the frozen refiner with a random-but-schema-valid proposer (or an adversarial one). If the gains persist, the scheduler alone carries the method; if they collapse, semantic edit quality is the load-bearing component. A complementary audit would have humans label refiner proposals for semantic correctness and test whether invalid proposals correlate with next-iteration performance drops.","tokens_in":23540,"feed_emoji":"🧰","tokens_out":5732,"duration_ms":52076,"temperature":0.7,"pith_summary":"The paper argues that the right role for skills in agentic reinforcement learning is not to serve as inference-time assets, but to act as a temporary, policy-aware training scaffold. PATS converts each rollout group into an evidence card, tracks per-task-type success rates, and chooses whether to expand, semantically revise, compress, or prune a natural-language context bank before the next rollout batch. The policy is optimized only by environmental rewards, and the scaffold is removed entirely at deployment. If the central claim holds, an agent can end up skill-free and still match skill-retaining state-of-the-art methods, with 25–50% lower deployment token cost, because the explicit guidance has been parameterized into the model rather than retrieved at test time.","feed_headline":"PATS matches skill-based agents using 25-50% fewer tokens","feed_subtitle":"A dynamic skill scaffold expands, revises, then removes itself, so the trained agent deploys with no external skills.","key_machinery":"The central mechanism is the policy-aware scaffold controller. It maintains a three-layer bank of natural-language entries (general principles, task-type procedures, mistake corrections), and for each task type it tracks an exponential moving average of success rate plus a global capacity pressure. These two signals select one of four edit modes—EXPAND, REVISE, COMPRESS, or FORCED_PRUNE—each with write budgets enforced by a deterministic validator. Rollout groups are condensed into evidence cards by a deterministic builder; a frozen 7B language-model refiner proposes schema-valid edits; and atomic commits make those edits visible only to the next rollout snapshot, so no batch consumes its ow","core_discovery":"On its own terms, PATS's central discovery is that training-effective support is not the same as inference-effective support. A fixed, high-quality skill library can raise immediate success while shrinking the success–failure variation inside rollout groups, which is exactly the contrast GRPO needs; PATS instead lets a controller schedule support so that weak policies receive concrete guidance, intermediate policies receive semantic revisions that track residual failures, and strong policies receive compression. The paper reports that across ALFWorld, WebShop, and seven search-augmented QA benchmarks, this schedule yields performance within about ±3% of state-of-the-art skill-based methods,","pith_inferences":["Inference: the same controller could schedule not just textual scaffolds but any auxiliary context—plans, rubrics, retrieved documents—by treating rollout success-rate EMA as a general curriculum signal; this is testable by swapping the scaffold content while keeping the scheduler fixed.","Inference: since only schema validity is checked deterministically, a natural stress test is to compare PATS against a random-but-valid refiner; the performance gap would isolate whether the gains come from scheduling or from the frozen refiner's semantic quality.","Inference: the likelihood-gap number is measured on a narrow two-gamefile diagnostic, so the broad claim of 'skill internalization' should be read as a mechanism hypothesis until replicated on a wider distribution—the paper itself flags this boundary.","Inference: if the intermediate-success-rate argument generalizes, then deliberately sampling tasks at ~50% success, or adapting scaffold support to hold groups near that contrast peak, could amplify GRPO signal beyond what PATS currently does; the paper explicitly does not target 50% success, so this is an open extension."],"forward_implications":["Deployment becomes skill-free by construction: the final policy is evaluated with the experience block removed, so any performance gain must be carried in the parameters, not in a retrieved library.","Token cost at inference drops 25–50% relative to skill-retaining baselines, because the scaffold contributes no prompt tokens once training ends.","The expand→revise→compress schedule gives a principled answer to when to add versus withdraw guidance: add when success is low, revise when failures shift, compress when competence is reliably high.","The offline-replay analysis implies that a static skill library can be a worse training curriculum even when it is a better inference aid, which reframes skill curation in agentic RL around rollout contrast rather than immediate utility."],"fun_headline_variants":["Training scaffold PATS matches SOTA with 25-50% fewer tokens","PATS adapts guidance per rollout, then discards it—matching SOTA","Policy-aware scaffold keeps weak agents on track, then steps aside","Skill-free training: PATS beats skill libraries using a dynamic scaffold"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the frozen 7B language model, guided only by evidence cards and mode instructions, proposes semantically valid and training-useful scaffold edits; schema checks catch format and duplication errors but not misleading content, so a bad proposal would inject harmful context into the next rollout.","fun_headline_variants_meta":{"raw":{"variants":["Training scaffold PATS matches SOTA with 25-50% fewer tokens","PATS adapts guidance per rollout, then discards it—matching SOTA","Policy-aware scaffold keeps weak agents on track, then steps aside","Skill-free training: PATS beats skill libraries using a dynamic scaffold"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000161,"raw_usage":{"total_tokens":1052,"prompt_tokens":700,"completion_tokens":352,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":444,"completion_tokens_details":{"reasoning_tokens":272}},"tokens_in":444,"tokens_out":352,"duration_ms":4300,"temperature":1.0,"reasoning_tokens":272,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T07:28:35.372666+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PATS with the same controller and budgets but replace the frozen refiner with a random-but-schema-valid proposer (or an adversarial one). If the gains persist, the scheduler alone carries the method; if they collapse, semantic edit quality is the load-bearing component. A complementary audit would have humans label refiner proposals for semantic correctness and test whether invalid proposals correlate with next-iteration performance drops.","supporting_citations":[],"review_version":1}