{"id":"7a10ba11-b07b-4bd6-b13d-584768de77e0","arxiv_id":"2608.05446","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"EvoHarness-RL trains a Qwen3-8B agent with supervised fine-tuning and cost-aware GRPO to selectively use a Belief-Progress-Experience external workspace, reaching 96.9% success on ALFWorld.","lead":"This paper trains an 8-billion-param LLM agent to decide when to read and write an external workspace that tracks beliefs, progress, and experience. On the ALFWorld household benchmark this reaches 96.9% success on seen tasks and 86.6% on unseen tasks, suggesting learned harness policies can beat hand-engineered scaffolds.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is not yet isolated: Table 2 ablates BPE only on the frozen inference-time harness, never on the trained EvoHarness-RL policy, so the +49.0 gain cannot be cleanly attributed to learned harness coordination rather than to SFT init, reward shaping, or the Claude-Opus-curated skill…","rationale":"The reader's weakest assumption is the dependence on a proprietary teacher and consolidation model. That is a valid generality concern, and it is related to the gap I identify: a Claude-Opus-curated skill bank could be the source of much of the value attributed to the trained policy. But the more immediate, internal problem is attribution: the paper never ablates BPE on the trained policy. Table 1 shows that the full SFT+GRPO pipeline reaches 96.9% and that standard GRPO reaches only 65.6%, but the GRPO control is not matched on SFT initialization, the shaped reward terms, or the availability of the BPE action space. The inference-time ablations in Table 2 demonstrate that the BPE interface helps a frozen Qwen3-8B, not that the trained policy's coordination decisions are what drive the headline gain. This matters because the paper's stated contribution is a trainable harness policy, not merely a curated memory system. The missing experiment is directly actionable: ablate BPE components on the final trained policy, and train a no-op-harness control with identical SFT and rewards. If the trained policy still succeeds without the harness state, the central claim should be substantially weakened. If the ablations show large drops, the claim is supported. The reader's CONDITIONAL verdict is appropriate; I would keep it conditional on this additional evidence. The paper is transparent about implementation details, which is a strength, but transparency does not substitute for the required control.","tokens_in":15187,"tokens_out":11250,"duration_ms":107390,"concrete_test":"Run the one-at-a-time BPE ablations on the final trained EvoHarness-RL policy, mirroring Table 2 for the trained agent: disable Experience (recall/note and the skill store), then Progress, then Belief, while keeping the trained policy weights and all other evaluation settings fixed. Also run a matched control trained with the identical SFT init and reward function but with harness actions made no-ops (one step consumed, no information returned). Report seen-split success over at least three seeds. If removing Experience or emptying BPE leaves success near 96.9%, the central attribution to learned harness coordination fails; if success drops substantially toward the 50-70% range in both checks, the claim is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that training a policy to construct and selectively access BPE external state produces large gains. The strongest direct evidence is the progression EvoHarness-Base (56.4) -> SFT (68.6) -> RL (96.9) in Table 1, together with the Table 2 component ablations. However, Table 2's bottom block ablates the three BPE components only on the frozen inference-time harness (EvoHarness-Base); the top block is labeled 'Reference: learned EvoHarness-RL policies' but no learned-policy ablation is reported. Section 3.3's conclusion that 'the absence of any single component significantly harms execution' is therefore established only for the prompt-time harness, not for the trained policy whose gains are the headline result. The comparison against standard GRPO (65.6) does not close this gap because it is not matched on SFT initialization, reward terms (efficiency, time-dependent diversity, spam/format penalties), or the harness action space. The reward schedule in Eqs. 5-6 itself contains an annealing curriculum, so the 'harness annealing' dynamic may be largely a consequence of the reward design. In addition, both the SFT demonstrations and the experience-store consolidation use Claude Opus (Appendix C), so a high-quality externally curated skill bank is available to the trained policy. Without a trained-policy ablation that removes BPE state (or at least the Experience store), the observed 96.9% could be driven by the curated skill content and reward engineering rather than by the learned 'when to read, update, or consolidate' coordination that the paper claims. This is the most load-bearing gap because it targets the attribution of the central claimed gain.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EvoHarness-RL, a two-stage training method that teaches an LLM agent to construct and selectively access an external harness state organized as Belief, Progress, and Experience (BPE). The agent interacts with this state through four meta-actions (track, commit, recall, note), and is trained first by supervised fine-tuning on teacher trajectories and then by cost-aware GRPO with a reward that combines task success, efficiency, diversity, and format penalties. On ALFWorld with Qwen3-8B, the method reportedly reaches 96.9% average success on the seen split and 86.6% on the unseen split, outperforming several memory-augmented and trainable baselines. The paper also reports two training dynamics: harness annealing (decreasing harness calls over RL) and harness evolution (growth and consolidation of the experience store). The central claim is that training a policy to coordinate with an external harness workspace yields large gains for long-horizon agents, beyond simply providing a fixed scaffold.","tokens_in":15601,"tokens_out":3020,"duration_ms":28562,"significance":"If validated, the central claim is significant: it suggests that the runtime policy for using memory, state trackers, and skill stores should itself be learned, rather than fixed by prompts or heuristics. The BPE abstraction is clean and well motivated, and the two-stage SFT-plus-cost-aware-GRPO recipe is a concrete, transferable design. The paper also provides useful analysis of how harness usage changes during training, and the ALFWorld instantiation is fully specified with hyperparameters in Table 4. However, the current evidence is not sufficient to isolate the claimed mechanism: the headline gain is demonstrated on a full system, while the component ablations are only run on the frozen inference-time harness, not on the trained policy, and the comparison baselines are not matched on initialization, reward terms, or backbone. The dependence on a proprietary teacher and consolidation model further limits the generalizability of the claim as stated.","major_comments":[{"comment":"Table 2 ablates the three BPE components only on the frozen inference-time harness (EvoHarness-Base), while the top block merely lists the learned policies without any learned-policy ablation. Section 3.3 concludes that 'the absence of any single component significantly harms execution,' but this is established only for the prompt-time harness, not for the trained EvoHarness-RL policy whose gains are the headline result. Without ablations that remove Belief, Progress, or Experience from the trained policy, the +49.0 improvement in Table 1 cannot be attributed to learned harness coordination rather than to SFT initialization, reward shaping, or the teacher-curated skill bank.","section":"Table 2 and Section 3.3"},{"comment":"The standard GRPO baseline (65.6%) is not matched to EvoHarness-RL: it is not initialized from the same SFT checkpoint, it does not receive the auxiliary reward terms in Eqs. (5)-(6), and it does not have the BPE action space. Therefore the comparison does not isolate the contribution of the harness policy: the gap could be driven by the extra reward shaping (efficiency, diversity, spam/format penalties) or by the SFT initialization. A controlled GRPO baseline that uses the same initialization and the same auxiliary rewards but without harness actions or harness state is needed to support the claim that harness coordination, rather than reward engineering, drives the gains.","section":"Table 1 and Section 2.4"},{"comment":"The reported harness-annealing dynamic may be largely a consequence of the reward design. The diversity weight lambda_div(u) in Eq. (6) is cosine-annealed over horizon U, and the efficiency bonus in Eq. (5) is described as 'naturally penalizing redundant harness queries.' Figure 3's decline in harness calls during GRPO is therefore expected from the curriculum, not necessarily an emergent internalization of harness use. The paper should either train with a fixed diversity weight or otherwise separate the effect of the annealing schedule from the policy's learned behavior.","section":"Section 4.1 and Eq. (6)"},{"comment":"The state-of-the-art comparison against SkillRL is not controlled: SkillRL is reported on Qwen2.5-7B while EvoHarness-RL uses Qwen3-8B, so the 7.0-point gap could reflect the larger or more recent backbone rather than the method. In addition, no error bars or multiple-seed variances are reported for any method, so it is unclear whether the reported differences are statistically stable. The SOTA claim requires a matched-backbone comparison and at least a small number of seeds.","section":"Table 1"},{"comment":"The method depends on a proprietary teacher and consolidation model: Claude Opus is used for SFT trajectory collection and for experience-store consolidation. This is a boundary condition on the central claim: the gains may not transfer to settings where only open-weight models are available, and no ablation varies the teacher or consolidation model. The paper should either provide such an ablation or explicitly qualify the claim that EvoHarness-RL teaches an open-weight agent to self-evolve its harness.","section":"Appendix C"}],"minor_comments":[{"comment":"The conclusion begins with 'We introduced We introduced EvoHarness-RL'; the duplicated phrase should be removed.","section":"Conclusion"},{"comment":"The plots show a single training run with no error bars or confidence intervals, despite being used to support the annealing and evolution dynamics; reporting multiple seeds would strengthen these analyses.","section":"Figures 3 and 4"},{"comment":"The paper does not state how many seeds or runs the main results in Tables 1 and 3 are based on; adding this information is important for reproducibility.","section":"Section 3.1"},{"comment":"The SFT dataset is small (87 trajectories, 1,153 conversation pairs), and the paper does not report whether the 500 training games used for collection were randomly sampled or whether the distribution over the six task families affects the results; a brief sensitivity check would help.","section":"Appendix C"},{"comment":"No code or data are released, which makes it difficult to independently verify the reported harness dynamics and the exact reward normalization in Eq. (5); a public release or a detailed pseudocode would improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The core idea and the BPE abstraction are interesting, and the ALFWorld results are potentially strong, but the current manuscript does not yet isolate the central claim. The missing trained-policy ablations, unmatched GRPO baseline, reward-curriculum confound, and backbone mismatch in the SkillRL comparison are all fixable with additional experiments, so I view this as a major-revision rather than a rejection. Given the paper's emphasis on learning rather than on a specific benchmark result, the authors should also consider whether the dependence on Claude Opus for SFT and consolidation is adequately discussed in the abstract and introduction, as it substantially qualifies the headline claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"EvoHarness-RL is a clear, well-executed empirical paper on a real problem: how an agent learns when to write and read external state instead of relying on hand-crafted prompts. The BPE abstraction (Belief, Progress, Experience) and the four meta-actions (track, commit, recall, note) are a sensible compact interface, and the two-stage SFT+GRPO recipe is a natural way to train it. The headline result—96.9% on ALFWorld seen, 86.6% unseen with Qwen3-8B—is strong, and the paper is careful to separate prompt-time, SFT, and RL variants.\n\nThe main weaknesses are about attribution and comparison. Table 2 ablates the BPE components only on the frozen inference-time harness, not on the trained policy that produces the headline gain. The standard-GRPO baseline (65.6) is not matched on SFT initialization, reward shaping, or the harness action space, so the +49.0 cannot be cleanly credited to learned harness coordination. The reward schedule in Eq. 6 includes a cosine-decayed diversity bonus with an explicit annealing horizon U; the 'harness annealing' dynamic in Figure 3 may be largely a consequence of that design rather than an emergent discovery. Both SFT demonstrations and experience consolidation use Claude Opus, and there is no ablation varying the teacher or replacing the consolidator with an open-weight model. The SkillRL comparison is on a smaller backbone (Qwen2.5-7B versus Qwen3-8B), which weakens the SOTA claim. And there are no error bars or released code.\n\nNone of these are fatal to the paper's value. The BPE interface and the training recipe are a genuine new combination, and the direction—learning when to use external state—is likely to be useful beyond ALFWorld. But the scientific claim about the mechanism needs a trained-policy ablation and a matched GRPO baseline to be solid. As is, it reads like a strong system paper with a slightly oversold interpretation.\n\nThis paper is worth a serious referee. I'd send it to review, and I'd ask for the missing ablations before acceptance. For my own reading group, it's a maybe: good for a discussion of agent harnesses, but not a must-read.","headline":"A strong empirical system paper whose headline gain is not yet cleanly attributed to learned harness coordination; worth refereeing with requests for a trained-policy ablation and matched baselines.","tokens_in":16199,"tokens_out":2859,"would_cite":true,"duration_ms":23495,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training an agent to decide when to read and write external state lifts ALFWorld success from 47.9% to 96.9%.","keywords":["LLM agents","long-horizon tasks","harness policy learning","external state","Belief Progress Experience","GRPO","reinforcement learning","ALFWorld"],"falsifier":"Train EvoHarness-RL on ALFWorld with an open-weight teacher and consolidation model in place of the proprietary frontier model, keeping everything else fixed; if the seen-split success rate falls back toward the 68.6% SFT-only level—or below the best frozen memory baselines—then the frontier teacher, not the learned cost-aware coordination policy, is carrying the reported 49-point gain.","tokens_in":15009,"feed_emoji":"🤖","tokens_out":9073,"duration_ms":70105,"temperature":0.7,"pith_summary":"EvoHarness-RL claims that the missing ingredient in long-horizon LLM agents is not more tools or bigger memories, but a trainable policy that decides when to form and consult external state. The paper exposes that state as Belief, Progress, and Experience (BPE), lets the agent act on it with four compact meta-actions—track, commit, recall, note—and trains the policy in two stages: supervised fine-tuning on teacher trajectories, then cost-aware GRPO that pays for each harness call out of the same step budget as environment actions. On the ALFWorld benchmark with a Qwen3-8B backbone, the resulting agent reaches 96.9% success on seen tasks and 86.6% on unseen tasks, a 49-point absolute jump over base ReAct and roughly the level of frontier models. The paper also reports two training dynamics: harness annealing, where the policy internalizes routine scaffold usage and becomes selective, and harness evolution, where the external experience store is consolidated into a compact, task-adaptive skill bank. If correct, this makes harness use a learned decision rather than a hand-engineered scaffolding convention across long-horizon tasks.","feed_headline":"A learned harness policy lifts ALFWorld success from 47.9% to 96.9%","feed_subtitle":"A Qwen3-8B agent that learns when to track, recall, and commit state beats frozen baselines and matches frontier models.","key_machinery":"The load-bearing object is the BPE workspace (Belief, Progress, Experience), a unified harness state rendered at each step as $H_t = (B_t, P_t, E_t)$: Belief tracks what is true in the environment, Progress records committed subgoals and their status, and Experience holds cross-episode skills and search priors. The agent interacts with this workspace through four harness meta-actions—track to read belief, commit to write progress, recall to retrieve experience, and note to record new insight—and both environment and harness actions draw from the same step budget. The second mechanism is a two-stage training recipe: supervised harness fine-tuning bootstraps the action protocol from teacher demonstrations, and cost-aware GRPO with a reward combining success, efficiency, action diversity, repetition penalties, and format validity learns selective, cost-aware coordination. An environment adapter grounds BPE in ALFWorld as a rule-based world-state parser, a bounded subgoal list, and a capacity-bound skill store with LFU eviction and periodic consolidation by an external summarizer model.","core_discovery":"On the paper's own terms, the central discovery is that external execution support becomes substantially more valuable when the agent is trained to control it. EvoHarness-RL abstracts heterogeneous runtime components—state trackers, subgoal records, skill banks—behind a single policy-facing interface $H_t = (B_t, P_t, E_t)$, navigated by four harness actions that consume the same interaction budget as environment actions. Supervised harness fine-tuning teaches Qwen3-8B the semantics of the action protocol, and cost-aware GRPO then optimizes when to read, write, and consolidate the workspace, with task success as the gatekeeping reward and efficiency, diversity, spam, and formatting terms shaping harness usage. Trained this way, EvoHarness-RL reports 96.9% average success on the ALFWorld seen split and 86.6% on the unseen split, outperforming both frozen memory baselines and trainable skill-learning agents, and matching or exceeding prompting-only frontier models. The paper interprets the trajectory of training as evidence that harness use anneals into selective access while the experience store evolves into a compact task-adaptive substrate.","pith_inferences":["Editorial inference: if the BPE abstraction is what enables transfer, the same interface could be adapted to web navigation or software engineering with a different environment adapter; the paper only instantiates ALFWorld, so this is an extension, not a claim.","Editorial inference: the annealing dynamic suggests a general curriculum—start with abundant scaffolding, then let RL prune it—that may apply to other forms of tool use, but the paper does not test this outside ALFWorld.","Editorial inference: because the teacher and consolidation model are both a proprietary frontier model, the paper does not establish whether the method's gains survive with open-weight teachers or rule-based consolidation; an ablation varying those components would decide this.","Editorial inference: the persistence of recall over commit and note after annealing suggests that cross-episode experience is the component whose value survives internalization, which would predict that environments with strong reusable priors benefit most from this method."],"forward_implications":["Harness use becomes a learned policy decision: after GRPO, the agent stabilizes near one harness call per episode instead of scaffold-like frequent calls, showing the policy internalizes recurring patterns.","The BPE interface provides gains across model scales at inference time: prompt-time harness adds +22.1 for GPT-4.1, +25.7 for GPT-5, and pushes Claude Opus 4.5 from 96.4% to 98.5%.","Training on top of SFT matters: EvoHarness-SFT reaches 68.6% seen and 69.4% unseen, while adding cost-aware GRPO reaches 96.9% seen and 86.6% unseen, so the coordination optimization, not just imitation, carries most of the final gain.","The three BPE components are synergistic: ablating any one component at inference time drops average success by roughly 6 to 8 points, with the largest drop when Experience is removed.","The learned policy generalizes beyond training tasks: the RL-optimized agent outperforms both the prompt-time harness and the SFT policy on the unseen split."],"supporting_citations":[{"why":"Supplies the ALFWorld environment and six task families on which all success rates are measured.","marker":"Shridhar et al., 2021"},{"why":"Provides the ReAct base policy that serves as the main baseline and the +49.0 improvement reference.","marker":"Yao et al., 2022"},{"why":"Contributes the GRPO algorithm used in the cost-aware coordination optimization stage.","marker":"Shao et al., 2024"},{"why":"Is the SkillOS trainable self-evolving baseline that EvoHarness-RL must beat (80.2% versus 96.9%).","marker":"Ouyang et al., 2026"},{"why":"Is the SkillRL trainable recursive-skill baseline that reports 89.9%, the closest strong comparator.","marker":"Xia et al., 2026"},{"why":"Motivates the Experience component by showing episodic reflection and reuse across attempts.","marker":"Shinn et al., 2023"},{"why":"Grounds the procedural-memory and skill-store design behind the Experience component.","marker":"Wang et al., 2024, 2025"},{"why":"Represents the harness-engineering line that EvoHarness-RL extends by making harness use a learned policy.","marker":"Jiang et al., 2026"}],"fun_headline_variants":["Self-evolving harness: Qwen3-8B learns to manage state, hits 96.9% on ALFWorld","Trainable harness control: Qwen3-8B hits 96.9% on ALFWorld","From 47.9% to 96.9%: A trainable harness policy on ALFWorld","Trainable harness beats fixed memory: Qwen3-8B scores 96.9% on ALFWorld"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the method's gains come from the learned coordination policy rather than from the quality of the proprietary frontier teacher that supplies demonstrations and consolidates the skill store; the paper does not vary either model.","fun_headline_variants_meta":{"raw":{"variants":["Self-evolving harness: Qwen3-8B learns to manage state, hits 96.9% on ALFWorld","Trainable harness control: Qwen3-8B hits 96.9% on ALFWorld","From 47.9% to 96.9%: A trainable harness policy on ALFWorld","Trainable harness beats fixed memory: Qwen3-8B scores 96.9% on ALFWorld"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001032,"raw_usage":{"total_tokens":4416,"prompt_tokens":1083,"completion_tokens":3333,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":699,"completion_tokens_details":{"reasoning_tokens":3219}},"tokens_in":699,"tokens_out":3333,"duration_ms":18997,"temperature":1.0,"reasoning_tokens":3219,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T12:56:11.845035+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train EvoHarness-RL on ALFWorld with an open-weight teacher and consolidation model in place of the proprietary frontier model, keeping everything else fixed; if the seen-split success rate falls back toward the 68.6% SFT-only level—or below the best frozen memory baselines—then the frontier teacher, not the learned cost-aware coordination policy, is carrying the reported 49-point gain.","supporting_citations":[{"cited_title":"Agent-BRACE: Decoupling Beliefs from Actions in Long-Horizon Tasks via Verbalized State Uncertainty","cited_arxiv_id":"2605.11436","evidence_quote":"Supplies the ALFWorld environment and six task families on which all success rates are measured."},{"cited_title":"Memp: Exploring agent procedural memory","cited_arxiv_id":null,"evidence_quote":"Is the SkillRL trainable recursive-skill baseline that reports 89.9%, the closest strong comparator."}],"review_version":1}