{"id":"72f59ba2-2bd2-4e0a-ae8a-bddb2313be7a","arxiv_id":"2608.06503","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Recurrent context compression can destabilize long-horizon agents, and a boundary-local verifier that tunes the compression prompt through paired rollouts improves reliability on AppWorld.","lead":"This paper studies how automatically summarizing an AI agent's earlier steps changes its later behavior. It finds that compression can make agents lose track of recent progress, causing repeated or blocked actions, and introduces a verifier-guided method that tunes the summary instructions using trial continuations, improving reliability on AppWorld.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The verifier's blocked/repeated-action proxy is never validated against task success, and final prompt selection uses terminal Pass2, so the reported gains may not demonstrate that boundary-local supervision is the cause.","rationale":"The paper is an honest, well-written preliminary study with a novel and falsifiable proposal. Its strongest evidence is the behavioral diagnosis in Section 3: matched-context probes show compression attenuates recent updates and increases blocked/repeated actions, and these patterns are plausible. The code, prompts, and baselines are documented in unusual detail. However, the central claim of the paper is not just that compression causes regressions, but that the boundary-local proxy is a better supervision signal than terminal outcomes. The current experiments do not isolate that claim. The final prompt is chosen by terminal Pass2 on the dev split, so the reported test gains could be produced by the selection stage even if the verifier were pure noise; equivalently, the proposer's system-prompt audit could be doing the work. The reader's weakest-assumption analysis (silent state corruption) is closely related, but the deeper issue is that no evidence links ΔG_b(s) to task success. The random-label ablation proposed above would settle this. If the ablation reproduces TRACE's gains, the boundary-local verifier is not causally important and the paper's framing should be revised; if it fails, TRACE's mechanism is supported. Because the authors explicitly label the study preliminary and the reader already requested conditional acceptance, I do not recommend changing the verdict: conditional acceptance requiring this causal validation, multi-run error bars, and ideally a second benchmark is the right call. I agree with the reader that the verifier proxy is the weakest assumption; my emphasis is on validating that proxy's alignment with task success rather than only its blind spot for silent corruption.","tokens_in":22116,"tokens_out":8150,"duration_ms":80433,"concrete_test":"Run a label-ablation of the optimization pipeline: use the same 12 boundary inputs, the same proposer prompt, the same downstream system-prompt audit, and the same dev-set selection among five candidates, but replace the verifier-derived preferences with random (or reversed) preference labels. Repeat the full procedure several times with different random seeds. If the random-label condition matches TRACE's dev and test Pass2 (~67.3 and ~77.1 respectively), the boundary-local verifier is not the cause of the improvement; if it degrades substantially, the preferences are causal. In the same experiment, record ΔG_b(s) and blocked/repeated counts for the selected TRACE template and the base prompt to verify that the optimized prompt actually reduces the regressions the verifier targets.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 defines the verifier score Q_b(s) = -ΔG_b(s), where ΔG_b(s) counts blocked or repeated actions over K=5 closed-loop steps. The paper's central claim is that this boundary-local signal is a better supervision signal than terminal outcomes. For that claim to hold, ΔG_b(s) must rank summaries by their effect on eventual task success. The paper never tests this alignment. Section 7 admits the verifier 'may not capture silent state corruption'; more importantly, Section 4.2's final template P* is selected by terminal Pass2 on the development split (Eqs. 9-10), not by verifier scores. With five candidates, two runs per task, and no significance tests or error bars, the best dev template can win by noise. The proposer is also given the downstream system prompt and an audit instruction, so TRACE's advantage over ACON could come from the audit or from terminal selection rather than from boundary-local preferences. No post-optimization verifier scores or blocked/repeated rates are reported for TRACE, so the proposed mechanism—fewer execution regressions yielding better task outcomes—is unverified. If the proxy is misaligned, the preference loop optimizes the wrong objective and the apparent gains are an artifact of the selection stage or proposer bias.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how recurrent context compression affects frozen long-horizon LLM agents. Using the AppWorld benchmark, it first presents diagnostic probes showing that replacing the working context with structured summaries increases blocked actions, repeated exploration, and run-to-run instability, especially on longer-horizon tasks. It then proposes TRACE, a framework that evaluates each compaction boundary by paired closed-loop rollouts from the same environment state, defines a verifier score Q_b(s) = -ΔG_b(s) based on blocked and repeated actions, and uses pairwise preferences between candidate summaries to have a frozen proposer revise a natural-language compression prompt. The final prompt is chosen by Pass2 on the development split and evaluated on the test split. On AppWorld, TRACE reports higher accuracy, Pass2, and Pass@2 than existing compression baselines, while keeping average steps near full-context execution, and the optimized prompt transfers to a second model (Kimi-K2.7-Code).","tokens_in":22382,"tokens_out":8243,"duration_ms":74562,"significance":"If the central claims hold, boundary-local evaluation would be an attractive way to optimize context compression for frozen agents, because it avoids modifying models and uses cheap, local signals instead of terminal rewards. The paper has several genuine strengths: the diagnostic studies use paired controls and bootstrap confidence intervals; the optimization protocol freezes all models; baseline prompts are taken from official repositories with recorded commits and hashes; and the authors explicitly acknowledge that the verifier may miss silent state corruption. These features make the empirical study more transparent than many similar papers. However, the main results are based on only two runs per condition with no error bars, and the reported gains are not yet causally attributable to boundary-local preferences because final prompt selection is end-to-end and the proposer also receives the downstream system prompt and an audit instruction. The significance of the paper therefore depends on additional validation steps that are feasible within the manuscript's scope.","major_comments":[{"comment":"The main empirical claims rest on two runs per condition with no confidence intervals or significance tests. The development-selection rule in Eq. (9) and the test tables report Pass2 and Pass@2 computed from the same two runs, so the headline gaps (e.g., TRACE over Prompting-O by 5.7 accuracy and 7.8 Pass2 in Table 1) are well within the range that could be produced by sampling noise. The diagnostic probes in Section 3 use task-level bootstrap intervals; the main tables should provide the same treatment, including intervals over the selection step, before the paper claims that TRACE 'consistently outperforms' existing baselines.","section":"§5.2, Tables 1-2; §4.2 Eq. (9)"},{"comment":"The verifier score Q_b(s) is never validated as a predictor of eventual task success. The paper's central claim is that boundary-local supervision is better than terminal outcomes, but to support that claim the paper must show that ΔG_b(s) ranks summaries by their effect on terminal success. The current design cannot establish this: final template selection uses terminal Pass2 on the development split (Eq. (10)), and, with five candidates and two runs each, the selected template may win by noise. The paper also reports no post-optimization verifier scores or blocked/repeated rates, so the proposed mechanism is unverified. Section 7 explicitly concedes that the verifier may not capture silent state corruption, which is a limitation of the optimization signal itself, not only of the evaluation. A concrete fix would be to report the correlation between verifier preferences and Pass2 across the five candidate prompts, or to evaluate a version selected solely by verifier score.","section":"§4.1-§4.2, Eqs. (2)-(4), (10); §7"},{"comment":"The proposer receives the downstream system prompt and an explicit 'audit-first' instruction in addition to the preference pairs. The optimization therefore does not isolate the contribution of boundary-local preferences: TRACE's advantage over ACON could come from the system-prompt-aware audit, from the richer input (including the downstream contract), or from the end-to-end selection stage, rather than from the verifier. The paper needs an ablation that removes one component at a time, for example a proposer given the audit instruction but no preference pairs, or preference pairs without the downstream contract, to attribute the improvement.","section":"§4.2, System-aware prompt proposal; Appendix B.2"},{"comment":"The optimization signal is built from only 12 boundaries, selected because they exhibit blocked actions, and only three candidate summaries per boundary. This yields a small, non-random preference set, and the selected prompt may overfit these boundaries. Since the final claim concerns general compression reliability, the paper should report how the 12 boundaries were sampled, how many boundaries exhibited blocked actions in total, and the sensitivity of the final template to the choice of training boundaries.","section":"§4.2, Eqs. (5)-(7)"}],"minor_comments":[{"comment":"TRACE (58.3) is below LLMLingua-2 (62.5) on Medium Pass2, in tension with the 'consistently outperforms' phrasing in the Introduction and Abstract; please qualify this claim once intervals are added.","section":"Table 1, Medium Pass2 row"},{"comment":"The value of K in Eq. (2) is never stated; Section 3.3 uses five actions for the diagnostic rollouts, but the verifier's K should be specified explicitly.","section":"§4.1, Eq. (2)"},{"comment":"The transfer claim is based on a single target model and the same two-run protocol; the sentence 'broader cross-model generalization remains to be established' is appropriate, but the abstract's 'transfers to a second model' should be reported with intervals or with a caveat about the two-run basis.","section":"§5.4, Table 2"},{"comment":"B is called a 'context budget' but is compared with |...| in tokens; specify the unit and how the budget is measured for different models' tokenizers.","section":"§2.2, Eq. (1)"},{"comment":"The proposer prompt is very long and contains task-specific details; a condensed description in the main text would help readers, with the full prompt left to the appendix.","section":"Appendix B.2"}],"recommendation":"major_revision","confidential_remarks":"The main claim is plausible but not yet isolated: the verifier proxy needs validation against task success, the proposer audit is a confound, and the main tables need seed-level intervals. The diagnostic study is well executed and the paper is honest about its limitations, so the issues appear addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this for the diagnosis and the boundary-local evaluation idea; the performance claims are not yet solid. The paper's best part is the behavioral analysis: Figure 3 shows compression attenuates recent-interaction effects, and Figure 4 shows a marginal increase in blocked actions and refetch/replay at the first five post-compaction steps. Those probes are controlled, use bootstrap CIs, and genuinely isolate a failure mode that trajectory-level methods miss. The TRACE framework is a reasonable next step—paired closed-loop rollouts from the same state are a cleaner supervision signal than terminal success/failure. I also credit the authors for stating the verifier's limitation (Section 7: silent state corruption) and for shipping code.\n\nThe soft spots are exactly where the stress-test note points. First, Tables 1 and 2 are two runs per condition with no error bars or significance tests. The abstract's 'consistently outperforms' is already contradicted by the medium Pass2 cell (LLMLingua-2 62.5 vs TRACE 58.3) and the transfer Pass@2 (89.9 vs 91.7 for full context). Second, and more important, the verifier score is never validated against task success. The final template is chosen by terminal Pass2 on the dev split (Eqs. 9-10), not by verifier scores. With five candidates and two runs, the selected template can win by noise. The proposer also receives the downstream system prompt and an audit instruction, so TRACE's advantage could come from that richer input rather than from boundary-local preferences. The paper does not report post-optimization verifier scores or blocked/repeated rates, so the proposed mechanism is unverified. These are addressable, but they are load-bearing for the causal claim.\n\nThe paper is honest, clearly written, and the central idea is not obviously wrong. It is a preliminary study on one benchmark. I would send it to peer review because the diagnosis and the boundary-local verifier deserve scrutiny and the community would benefit from the discussion. But I would ask for multi-run error bars, significance tests, a second benchmark, and ideally a proxy-validation experiment before accepting. For my own work, I'd cite the diagnostic findings and the method even if the empirical claims need strengthening.","headline":"A genuinely novel boundary-local verifier for compression prompts, but the headline results rest on two runs without error bars and the proxy is never validated against task success.","tokens_in":22917,"tokens_out":2908,"would_cite":true,"duration_ms":26049,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Recurrent context compression degrades long-horizon agents by weakening the influence of recent interactions, and TRACE shows that counting the resulting blocked and repeated actions at each compaction boundary is a better supervision…","keywords":["context compression","long-horizon agents","execution instability","blocked actions","repeated exploration","verifier-guided prompt optimization","multi-run reliability","AppWorld"],"falsifier":"Take a set of compaction boundaries and hand-edit candidate summaries so that one summary silently changes a value the agent will later use, such as an ID, amount, or date, while another keeps it faithful, ensuring neither produces a blocked or repeated action in the rollout; if the verifier scores the corrupt summary as well as or better than the faithful one, the boundary-local signal misses the failure mode the paper concedes it may miss. A stronger version is to construct or select tasks whose failures are driven by such silent corruption and check whether TRACE's advantage over baselines shrinks on them.","tokens_in":1818,"feed_emoji":"🤖","tokens_out":2891,"duration_ms":101595,"temperature":0.7,"pith_summary":"Recurrent context compression, which periodically replaces a growing agent context with a shorter summary, is usually judged by whether the final task still succeeds. This paper argues that terminal judgments miss the real failure mode: compression weakens the influence of recent interactions, and the frozen agent then loses its place in the trajectory, producing blocked actions and repeated exploration. To make the failure measurable, the authors introduce TRACE, which evaluates each summary at the compaction boundary where it replaces raw history by running paired continuations from the same environment state and counting the extra blocked or repeated actions the summary causes. Those boundary-local preferences revise a natural-language compression template while every model stays frozen. On AppWorld the result is higher accuracy, higher Pass2 reliability, and near-full-context efficiency compared with existing compression baselines, plus a template that transfers to a second agent model.","feed_headline":"Blocked and repeated actions reveal what context compression breaks","feed_subtitle":"Scoring each compaction event by added blocked or repeated actions beats terminal-outcome baselines on AppWorld.","key_machinery":"The load-bearing object is the boundary-local execution verifier. At a compaction boundary, the verifier rolls out the frozen agent from the same environment state under two renderings: PRE, which keeps the pre-compaction context, and POST, which replaces the compressible history with a candidate summary. Each rollout's execution burden counts actions that are blocked by the environment's native error contract or whose canonicalized tool-call signature repeats a call made before the boundary or earlier in the same continuation, and the verifier score is the negative expected increase in that burden. The score never reaches the proposer as a number: it only orders summaries at the same boundary into better and worse pairs, and the proposer sees the compressor inputs, those pairs, and the downstream system prompt, but not the rollouts. The final template is then chosen by end-to-end Pass2 on the development split, so terminal outcomes serve model selection rather than per-summary supervision.","core_discovery":"At its center, the paper claims that recurrent context compression degrades long-horizon agents not mainly by deleting facts but by breaking the agent's local sense of where it is in the task: what has been completed, what is actionable, and whether it may stop. It supports this with three probes on AppWorld: summary-conditioned agents terminate in the required form less often than FIFO- or full-history-conditioned agents at the same decision point; compressing a recent interaction weakens its behavioral effect relative to keeping it verbatim; and closed-loop rollouts after compaction show more blocked and error actions and more refetching or replaying of previously executed calls. From this diagnosis TRACE builds a training signal: at each boundary, paired rollouts from the same state, with and without the candidate summary, measure the compression-induced execution burden, and the induced preferences over summaries become the feedback for a frozen proposer rewriting the compression prompt. With this prompt, TRACE reaches 77.1 average accuracy and 67.3 Pass2 on AppWorld test-normal, versus 71.4 and 59.5 for the best existing compressed baseline, keeps average steps close to the full-context run, and the same template applied to a second model exceeds full-context accuracy and Pass2. The authors describe these results as early evidence for boundary-local evaluation rather than a settled claim.","pith_inferences":["A natural testable extension is to replace the blocked and repeated action proxy with a richer boundary signal, such as the divergence between next-action distributions or a state-consistency check, and see whether gains grow on tasks whose failures are silent.","The same boundary-local preference signal could serve as a dense reward for training learned compressors, not just for prompting a frozen proposer; the paper optimizes only a natural-language template and leaves that route implicit.","The paper's observation that recency-only truncation is competitive at moderate budgets in a refetchable environment suggests a hybrid policy that keeps recent turns verbatim and summarizes only older state might capture much of TRACE's benefit at lower cost.","If boundary-local evaluation generalizes beyond compression, it could become a general diagnostic for context-management choices such as memory tiers and retrieval, by measuring whether any replacement of history changes future action distributions."],"forward_implications":["Compression quality for long-horizon agents should be evaluated at the boundary where the context is replaced, not only by the terminal task outcome, because a defective summary can be masked by later recovery and a failed run can be unrelated to any particular summary.","Optimizing a compression prompt against blocked and repeated actions improves multi-run reliability as well as average accuracy, meaning the added signal reduces the run-to-run instability that single-run metrics miss.","The optimized template's transfer to a second agent model without further tuning implies that good compression policy is at least partly model-independent and can be reused across agents.","Because TRACE holds step count near full-context while cutting peak token use, reducing compression-induced regressive exploration is compatible with, and not opposed to, context efficiency.","Boundary-local preference feedback outperforms trajectory-level success and failure contrast on the same benchmark, suggesting that terminal labels are a noisier supervision channel per summary."],"supporting_citations":[{"why":"Supplies AppWorld, the stateful API-use benchmark that provides the tasks, environment, and evaluation splits for the behavioral diagnosis and all TRACE experiments.","marker":"(Trivedi et al., 2024)"},{"why":"Provides the ACON baselines and the natural-language compression-prompt optimization setting that TRACE extends with boundary-local feedback.","marker":"(Kang et al., 2026)"},{"why":"Provides the recurrent compaction loop used as the compression harness and the Prompting-O structured-summary baseline.","marker":"(OpenClaw Contributors, 2026a)"},{"why":"Defines the P@k and Pk repeated-run reliability metrics used to show that compression converts reliably solved tasks into intermittently solved ones.","marker":"(Yao et al., 2025)"},{"why":"Extends the repeated-run reliability metric tradition and motivates Pass2 as the selection target for the final template.","marker":"(Barres et al., 2025)"},{"why":"Provides LLMLingua-2, the token-pruning baseline that helps separate information loss from execution-state mislocalization.","marker":"(Pan et al., 2024)"},{"why":"Specifies MiniMax-M3, the model used as compressor, downstream agent, and proposer in the main experiments.","marker":"(MiniMax, 2026)"},{"why":"Specifies Kimi-K2.7-Code, the second model used to test whether the optimized compression template transfers.","marker":"(Moonshot AI, 2026)"}],"fun_headline_variants":["Context compression breaks agents' sense of where they are in a task","TRACE uses paired rollouts to score each summary and reduce errors","Why compression causes blocked actions and repeated exploration","Verifier-guided compression outperforms baselines on AppWorld","Paired rollouts from the same state spot bad summaries"],"cache_read_input_tokens":25088,"weakest_assumption_plain":"The method's whole optimization signal rests on the premise that compression damage shows up at the boundary as blocked or repeated actions; the paper itself concedes that silent state corruption might produce neither, and if that failure mode dominates, the verifier's preferences would not reward fixing it.","fun_headline_variants_meta":{"raw":{"variants":["Context compression breaks agents' sense of where they are in a task","TRACE uses paired rollouts to score each summary and reduce errors","Why compression causes blocked actions and repeated exploration","Verifier-guided compression outperforms baselines on AppWorld","Paired rollouts from the same state spot bad summaries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00062,"raw_usage":{"total_tokens":2873,"prompt_tokens":941,"completion_tokens":1932,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":1850}},"tokens_in":557,"tokens_out":1932,"duration_ms":17580,"temperature":1.0,"reasoning_tokens":1850,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:18:24.153535+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of compaction boundaries and hand-edit candidate summaries so that one summary silently changes a value the agent will later use, such as an ID, amount, or date, while another keeps it faithful, ensuring neither produces a blocked or repeated action in the rollout; if the verifier scores the corrupt summary as well as or better than the faithful one, the boundary-local signal misses the failure mode the paper concedes it may miss. A stronger version is to construct or select tasks whose failures are driven by such silent corruption and check whether TRACE's advantage over baselines shrinks on them.","supporting_citations":[],"review_version":1}