{"id":"dfaa7ccb-4dc2-4e09-b29f-71fee6547737","arxiv_id":"2608.01964","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An external task-state harness with manager-executor-auditor loops lifts Qwen 3.7-Plus from 51.8% to 80.7% on WeaveBench and from 69.7% to 77.2% on Terminal-Bench 2.1 under matched backends.","lead":"This paper introduces a manager-executor-auditor harness that stores task state outside the agent context and updates it only from independently verified environment facts. It reports large gains on long-horizon agent benchmarks, suggesting that harness design matters as much as model choice.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Load-bearing premise is auditor accuracy: the manager persists only auditor-verified facts, yet no auditor error rates are reported and the paper itself concedes 'confidently verified wrong answer' cases.","rationale":"The reader identified the same load-bearing assumption, and I agree. The reason it is load-bearing rather than a peripheral robustness concern is structural: Eq. (1) makes S_{i+1} a function of T, S_i, V_i, with V_i produced solely by Φ_aud. There is no second source of truth in the loop; the harness's core novelty is exactly that audit reports are the only cross-round memory (Fig. 1). Therefore the correctness of the whole state-management claim is conditional on the conditional correctness of Φ_aud. The paper's matched WeaveBench and Terminal-Bench gains are real evidence that the architecture helps, but they do not by themselves establish that the help comes from independently verified facts; an executor self-report plus fresh-context replanning baseline would be needed. That is why the missing auditor-error analysis is the decisive gap. The OSWorld hybrid-tool confound is secondary because the WeaveBench and Terminal-Bench comparisons are matched. The internal B.2.2 admission strengthens the concern: the authors themselves observe verified wrong answers. The concrete test—measuring auditor verdict accuracy against ground truth—would settle whether this is a rare edge case or a systematic weakness. If the false-positive rate is low, the CONDITIONAL verdict can move to ACCEPT; if high, the central claim is not supported as stated.","tokens_in":31536,"tokens_out":6503,"duration_ms":77416,"concrete_test":"Select a random sample of 30 WeaveBench and 20 OSWorld 2.0 tasks. Instrument LongHorizon-Harness to log, after each MEA round, the auditor's completion status, integrity status, and every fact record marked 'verified'. At task end, compare each 'complete/clean' verdict and each verified fact against the official benchmark judge and ground-truth artifact checks (e.g., file hashes, GUI state, hidden tests). Compute the false-positive rate—audit certifies a state that the judge later scores <1 or invalidates—and the false-negative rate. If the completion-claim false-positive rate exceeds ~5%, the 'only independently verified facts' premise is materially violated, and the central causal claim needs an auditor-accuracy bound or an oracle-auditor ablation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the MEA loop's guarantee that persistent task state advances only on 'independently verified' audit facts (§2.2, Eq. 1). Because the manager has no environment access, every subsequent contract is built from previous audit reports v_i; if the auditor certifies a false completion or an invalid fact, that error becomes the premise for all later planning—an externalized, confidently verified wrong answer rather than context rot. The paper provides no measurement of auditor precision/recall, and the auditor is the same backbone model as the executor in all reported runs (§3.1), so the independence is procedural, not cognitive. The manuscript itself concedes the failure mode: Appendix B.2.2 says 'plausible evidence can be mistaken for completed evidence' and 'a misinterpreted contract can still lead to a confidently verified wrong answer.' If audit false positives are non-negligible, the observed gains could be due mainly to task decomposition and fresh-context execution, not to the paper's claimed verified-state mechanism. The OSWorld hybrid-tool confound is real but secondary; matched WeaveBench and Terminal-Bench results still need the auditor premise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LongHorizon-Harness, a harness that separates long-horizon task execution from task-state management via a Manage-Execute-Audit (MEA) loop. A manager maintains an explicit task-state ledger and proposes one bounded subtask per round; an executor performs the subtask in a fresh context; a read-only auditor inspects the environment and produces an audit report that is the only cross-round memory. The paper reports large improvements over matched baselines for Qwen 3.7-Plus on WeaveBench (51.8→80.7 PassRate), Terminal-Bench 2.1 (69.7→77.2), and an OSWorld 2.0 subset with Claude Opus 4.7 (20.6→35.3), plus a smaller OSWorld 2.0 gain (2.8→8.3) in a hybrid GUI+CLI setting. It also provides domain/tag-level breakdowns and qualitative case studies.","tokens_in":31890,"tokens_out":7771,"duration_ms":79556,"significance":"If the results hold, the paper makes a significant empirical case that the harness—not just the model—is a primary determinant of long-horizon agent performance. The matched WeaveBench and Terminal-Bench comparisons are strong: the same backbone and execution backend differ only by the MEA layer. The per-domain, per-tag, and per-task analyses are useful, and the case studies illustrate the intended failure-recovery mechanisms. The main scientific risk is that the central mechanism—'independently verified facts'—rests on an unmeasured auditor, and the flagship OSWorld comparison confounds tool access with state management. The paper is honest about residual failure modes, but those admissions also underline the load-bearing premise.","major_comments":[{"comment":"The load-bearing premise is that the read-only auditor produces reliable verdicts, since the manager's state update in Eq. (1) consumes only audit reports. No auditor precision/recall is reported, and in all runs the auditor is the same backbone as the executor (§3.1), so 'independent' is procedural, not cognitive. Appendix B.2.2 concedes 'plausible evidence can be mistaken for completed evidence' and 'a misinterpreted contract can still lead to a confidently verified wrong answer.' Please add a quantitative audit-error analysis (e.g., compare audit verdicts against benchmark ground truth on a sample) or an ablation with a different/stronger auditor. Without this, the gains could be attributed to task decomposition and fresh-context execution rather than to verified-state management.","section":"§2.4, Eq. (3); Appendix B.2.2"},{"comment":"The OSWorld 2.0 comparison is confounded: the baseline is the official single-action GUI setting, while LH-Harness uses a hybrid GUI+CLI tool pool (§A.2). The 2.8→8.3 gain therefore conflates added CLI tool access with the MEA loop. Figure 1's caption even claims improvement 'with the same backbone and execution backend' across all three benchmarks, which is not true for OSWorld. Please add a matched baseline (same hybrid tools without MEA, or a single-action LH-Harness) and correct the caption. This does not undercut the WeaveBench/Terminal-Bench results, but it weakens the OSWorld-specific and Fig. 4 cost-frontier claims.","section":"Table 2; Fig. 1; §A.2"},{"comment":"Evaluation reporting needs tightening. WeaveBench and OSWorld appear to be single runs; please report variance/confidence intervals or explicitly state that the benchmarks are deterministic under the evaluation protocol. Table 1's note says our runs use root privileges while official results use a regular user; please state explicitly whether the Qwen 3.7-Plus Claude Code baseline row (51.8%) was run under the same root privileges and the same judge configuration as LH-Harness, since this is the matched comparison on which the main WeaveBench claim rests. For Terminal-Bench, Fig. 3 should clearly identify which values are externally reported versus this paper's runs.","section":"§3.1; Table 1; Fig. 3"},{"comment":"The mechanism claim would be strengthened by an explicit ablation against a simpler fresh-context decomposition baseline without an audit-based state ledger (e.g., manager + executor only, or a memory-summarization wrapper). The current matched baseline (Claude Code) already includes planning and subagents, so it is not clear that the explicit ledger, rather than bounded contracts, drives the gains. This is a local issue and can be addressed with an additional experiment or a careful discussion.","section":"§2.2, Eq. (1); §3.3"}],"minor_comments":[{"comment":"The model name is inconsistent: 'Qwen-3.7-Plus' in §3.1 versus 'Qwen 3.7-Plus' elsewhere. Please harmonize.","section":"§3.1 and throughout"},{"comment":"PassRate is defined as the fraction of tasks with score at least 0.8, while the main text says 'percentage of fully passed tasks.' Please reconcile the two definitions.","section":"Appendix A.1"},{"comment":"The text refers to 'CUA-Harness' in the Terminal-Bench setup; this appears to be a typo for LongHorizon-Harness.","section":"Appendix A.3"},{"comment":"The asterisk legend for externally reported metrics is incomplete. Please mark each externally reported value explicitly.","section":"Fig. 3"},{"comment":"The executor is called 'fresh-context' but receives prior audit reports referenced by the contract. Please clarify that freshness applies to the raw interaction trajectory, not to all information.","section":"§2.3"},{"comment":"The paper would benefit from providing the manager/executor/auditor prompt templates and the task-state schema, or a clear pointer to them in the public repository, since these details are central to reproducing the MEA loop.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The main scientific risk is the unmeasured auditor reliability. I would ask the authors to provide auditor error-rate measurements or a convincing ablation before acceptance. The OSWorld comparison should be relabeled as hybrid vs. single-action in all public claims, including the abstract and Figure 1; as written, it overstates the matchedness. The WeaveBench and Terminal-Bench results, if confirmed, are strong enough to support the central claim, so the issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuine harness result, not a model trick. The matched WeaveBench jump (51.8 to 80.7) and Terminal-Bench gain (69.7 to 77.2) are paired comparisons with the same model and execution backend, and they make the case that external task-state management changes long-horizon outcomes. The OSWorld 2.0 gain is real but not clean, and the whole loop leans on an auditor whose error rate is never measured.\n\nWhat's new: the MEA loop's combination—manager with no environment access, executor in a fresh context each round, read-only auditor whose reports are the only cross-round memory, plus an adapter that wraps Claude Code/Codex without changing their native loops. Components exist separately, but this package and the matched gains are new. The case studies are effective: they show failures getting externalized, apparent completion getting rejected, and evidence chains surviving across rounds. The cost analysis is honest: manager is cheap, auditor is the main overhead, and on Terminal-Bench the harness actually uses fewer tokens.\n\nSoft spots, in rough order of importance.\n\nFirst, auditor accuracy is load-bearing and unmeasured. The manager updates the ledger only from audit verdicts; if the auditor certifies something false, every later contract is built on it. The paper reports no precision/recall, and the auditor is the same backbone as the executor in all runs, so the independence is procedural, not cognitive. The authors concede exactly this in B.2.2: 'plausible evidence can be mistaken for completed evidence' and a misinterpreted contract can produce a 'confidently verified wrong answer.' That does not kill the result, but it means the claimed verified-state mechanism is not fully evidenced; part of the gain could come from decomposition and fresh context alone.\n\nSecond, the OSWorld comparison is confounded. The baseline is single-action GUI; LongHorizon-Harness adds a CLI tool pool, so tool access and state management are tangled. The 2.8 to 8.3 and Opus subset numbers should be treated as hybrid-tools-vs-GUI, not as a clean test of MEA. Fixable: add a hybrid baseline or a single-action LH variant.\n\nThird, no error bars. WeaveBench appears to be a single run per task, Terminal-Bench uses three, OSWorld one. Given cost, some variance reporting is needed.\n\nMinor: OSWorld's evaluator and user simulator use the same qwen3.7-plus configuration, which is worth a sensitivity check; code link lacks a commit hash.\n\nWho this is for: anyone building or benchmarking agent harnesses. The architecture idea is transportable and the matched Terminal-Bench/WeaveBench results justify referee time. I'd send it out with a request for auditor reliability analysis and a cleaner OSWorld protocol.","headline":"Real architecture result with matched WeaveBench/Terminal-Bench gains; OSWorld is confounded and the load-bearing auditor accuracy is unmeasured, but it deserves a serious referee.","tokens_in":32358,"tokens_out":3099,"would_cite":true,"duration_ms":38517,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that long-horizon agent capability is a property of the whole model–harness system, not the model alone, proved by an audited task-state loop that lifts a single model's WeaveBench pass rate from 51.8% to 80.7%.","keywords":["long-horizon agents","task-state management","Manage-Execute-Audit loop","independent verification","context rot","computer-use agents","agent harness","LLM agents"],"falsifier":"A concrete test: build a task where the executor produces a visually or textually plausible artifact that violates an acceptance criterion, and measure whether the auditor flags it. If auditor false-negative rates are high, or if replacing auditor verdicts with executor self-reports leaves performance unchanged, the audit step is not load-bearing and the gains must come from decomposition or fresh contexts instead.","tokens_in":31461,"feed_emoji":"🤖","tokens_out":4654,"duration_ms":47688,"temperature":0.7,"pith_summary":"The paper argues that the main obstacle to long-horizon agent tasks is not any single step but the loss and corruption of task state across many interdependent steps. It proposes LongHorizon-Harness, a system that keeps task state outside the execution context, runs each subtask in a fresh context, and lets a read-only auditor verify the environment before updating the state. On matched comparisons with the same backbone model, this Manage-Execute-Audit loop raises Qwen 3.7-Plus from 51.8% to 80.7% on WeaveBench, from 69.7% to 77.2% on Terminal-Bench 2.1, and from 2.8% to 8.3% on OSWorld 2.0. The result matters because it suggests that a large share of agent capability can be credited to the harness that organizes, verifies, and accumulates progress, not merely to the underlying model.","feed_headline":"Audited task state lifts long-horizon agents from 52% to 81%","feed_subtitle":"A manager-executor-auditor loop verifies each step outside the growing context, letting the same model nearly double its completion rate.","key_machinery":"The Manage-Execute-Audit (MEA) loop is the load-bearing mechanism: a manager owns persistent task state and writes bounded subtask contracts; a fresh-context executor performs one contract; a read-only auditor independently inspects the resulting environment and produces the only evidence allowed to update the state. The task-state ledger itself—records marked completed, pending, blocked, or untrusted with references to audit evidence—is the central object. The loop's design means the executor's interaction history is discarded each round, so only compact, verified state persists across rounds.","core_discovery":"LongHorizon-Harness reformulates long-horizon execution as task-state management. The manager maintains a structured ledger of requirements, artifacts, and facts; the executor performs a single bounded subtask in a fresh context; and a read-only auditor inspects the resulting environment and returns a report. Only audit-verified findings advance the persistent state, and the executor's raw trajectory is discarded after every round. Across WeaveBench, OSWorld 2.0, and Terminal-Bench 2.1, the same backbone model completes tasks at substantially higher rates when wrapped in this loop, and the gains transfer to another backbone. The paper's central claim is that explicit, externally audited stat","pith_inferences":["If the auditor's verdicts are the only gateway into persistent state, then auditor accuracy, not model capability, becomes the effective ceiling on what the harness can certify. The paper does not measure auditor false positives or false negatives, so a direct test would be to inject a known-wrong executor report and see whether the audit catches it.","Because manager, executor, and auditor are usually the same backbone model, the 'independence' of the audit is procedural (fresh context, read-only tools, no trajectory) rather than cognitive. Using a different, possibly stronger or more conservative model as auditor is a natural extension the paper leaves implicit.","The framework suggests that progress on long-horizon benchmarks may be attainable by reorganizing inference around state verification even without new model training—an implication that could be tested by applying the same MEA wrapper to more backbones and seeing whether the gains persist.","The failure analysis points to a next bottleneck: tasks whose correctness depends on hidden thresholds or semantics not recoverable from visible state can still end in confidently verified wrong answers, so verifier design (knowing what state is authoritative) may become the next limiting factor."],"forward_implications":["Agent capability is a property of the model–harness system: a stronger harness can raise task-level performance for a fixed model, and stronger models plus explicit state management give complementary gains (e.g., Qwen with the harness beats Claude Opus with the baseline Claude Code harness on the same WeaveBench games subset).","Long-horizon gains appear across GUI+CLI, desktop, and pure command-line domains, so the mechanism is not specific to visual computer use.","The harness mainly raises the failure floor: tasks at or near zero baseline completion recover to substantial scores, while already-solved tasks see smaller changes.","Token cost is not a fixed multiplier: it can rise (WeaveBench ~2.3x, OSWorld ~3.6x output tokens) or fall (Terminal-Bench 24% fewer tokens) depending on model and task, because manager overhead is small and the auditor is the main added cost.","Independent auditing can detect an incorrect result and trigger recovery, but cannot supply a missing primitive capability; the benefit concentrates where the bottleneck is long-horizon reliability rather than a single skill."],"fun_headline_variants":["Audit loop keeps long-horizon agents on track, raising success from 52% to 81%","Manager-executor-auditor loop boosts agent completion on long tasks","Fresh-context execution with verified state lifts agent performance","Explicit task state and auditor verification nearly double agent success"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the read-only auditor can reliably certify completion and integrity from environment inspection; the paper reports no measurement of auditor error rates, and because the auditor is usually the same model as the executor, its independence is procedural rather than cognitive—if the audit is wrong, the external state becomes a source of confidently verified wrong answers.","fun_headline_variants_meta":{"raw":{"variants":["Audit loop keeps long-horizon agents on track, raising success from 52% to 81%","Manager-executor-auditor loop boosts agent completion on long tasks","Fresh-context execution with verified state lifts agent performance","Explicit task state and auditor verification nearly double agent success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000851,"raw_usage":{"total_tokens":3573,"prompt_tokens":813,"completion_tokens":2760,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":2681}},"tokens_in":557,"tokens_out":2760,"duration_ms":22255,"temperature":1.0,"reasoning_tokens":2681,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T17:35:59.986777+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: build a task where the executor produces a visually or textually plausible artifact that violates an acceptance criterion, and measure whether the auditor flags it. If auditor false-negative rates are high, or if replacing auditor verdicts with executor self-reports leaves performance unchanged, the audit step is not load-bearing and the gains must come from decomposition or fresh contexts instead.","supporting_citations":[],"review_version":1}