{"id":"4d75264e-20b2-47db-8567-95f4fe868076","arxiv_id":"2607.11423","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ToFu beats Claude Code and OpenCode on SWE-bench Verified Pass@1 across three LLMs while using about 28% fewer tokens than Claude Code on average.","lead":"ToFu is an open-source agent harness that edits code, runs tools, and manages context more token-efficiently than Claude Code on SWE-bench Verified. It is meant both as a practical research assistant and as a white-box baseline for studying harness design.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The dual research-assistant claim rests on coding benchmarks plus a thin, size-inconsistent human study; that proxy is the softest load-bearing support.","rationale":"The reader's strongest claim is correctly grounded in Table 1: full SWE-bench Verified, three backbones, better Pass@1 and fewer tokens vs Claude Code. That coding-efficiency result does not appear internally contradicted by the reported numbers. The reader's weakest assumption matches the softest support for the dual pitch: coding success plus a limited human study are asked to carry 'practical research assistant' and general harness-research baseline. The main-text vs Appendix A participant-count inconsistency (3 vs 7) and the absence of token/cost metrics outside coding reinforce that the proxy is under-specified, not that Table 1 is wrong. Cost language in abstract/intro is also not uniform in Table 1 (ToFu is not always cheapest), which the reader already flags. No stronger load-bearing flaw (e.g., grading error or impossible numbers) is evident from the manuscript. Verdict should stay CONDITIONAL: accept the coding head-to-head under the reported setup; require broader evaluation, consistent human-study reporting, and tighter cost claims before full acceptance as a dual research-assistant + harness baseline paper. Agreement with the reader is full on the load-bearing concern.","tokens_in":12122,"tokens_out":648,"duration_ms":6914,"concrete_test":"Run a fixed set of 20 non-coding research-assistant tasks (e.g., Overleaf section rewrite + compile, paper-reader grounded Q&A, multi-source literature comparison) with identical Claude Opus 4.6 backbone for ToFu vs Claude Code; report Pass/preference, avg tokens, and cost. If ToFu's token/cost edge disappears or preference falls near chance, the dual research-assistant claim weakens while coding results can stand.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's dual claim is that ToFu is both a practical research assistant (token-efficient, lower-cost, multilingual, usable product surface) and a white-box harness-research baseline with strong benchmarks. Table 1 solidly supports the coding half: higher Pass@1 than Claude Code and OpenCode across three models, with ~28.4% fewer tokens than Claude Code. The research-assistant half is much thinner. Limitations admit evaluation mainly focuses on coding; broader scenarios are a small human preference study. Main text says three participants; Appendix A reports seven annotators on 100 tasks (700 pairwise comparisons) with Claude Opus 4.6 only, and no head-to-head token/cost metrics for those tasks. Multilingual gains (Fig. 3) are also only MAPS:SWE-bench coding. So the dual claim treats SWE-bench success + one preference study as adequate for 'practical research workflows' and a general harness baseline. That is the least secure condition: if non-coding research workflows do not show the same efficiency/usability edge, the dual pitch overreaches even if coding numbers hold.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper presents TOFU, an open-source (MIT) agentic harness for research workflows that combines a modular architecture (UI, orchestration core, model abstraction, capability runtime, state/knowledge, context management) with three-layer context compaction, BM25 memory retrieval, multilingual translate-then-reason, and a dependency-aware swarm scheduler. Empirically, on the full 500-instance SWE-bench Verified suite graded by the official harness in udocker, TOFU reports higher Pass@1 than Claude Code and OpenCode across Claude Opus 4.6, GLM 5.1, and DeepSeek-v4-pro, with ~28.4% fewer average tokens than Claude Code (Table 1). Multilingual gains are shown on MAPS:SWE-bench (Fig. 3), and Appendix A reports a blind human preference study favoring TOFU. The dual claim is that TOFU is both a practical research assistant and a white-box baseline for harness research with application-level UX.","tokens_in":12378,"tokens_out":793,"duration_ms":7129,"significance":"If the coding results hold under independent reproduction, the work is a useful systems contribution: an open, inspectable harness that improves Pass@1 while reducing tokens relative to a strong commercial baseline across three backbones is rare and directly relevant to harness-evolution research. Strengths include full-suite SWE-bench Verified evaluation with official grading, multi-backbone comparison, open-source release, and a concrete design comparison to OpenCode (Table 3). The dual positioning as research assistant plus research object is timely for the community studying harness design, provided the non-coding claims are scoped to the evidence actually provided.","major_comments":[{"comment":"Abstract and §1 claim superior token efficiency, lower cost, and practical research-assistant support relative to existing harnesses. Table 1 supports higher Pass@1 and lower average tokens vs Claude Code, but cost is not uniformly lower (Claude Opus 4.6: $5.13 vs $4.97; OpenCode is cheaper on two backbones). The dual research-assistant claim therefore overstates cost superiority and rests mainly on coding benchmarks. Limitations already note that broader research-assistant evaluation is limited; the abstract/intro should be tightened to match Table 1 (token efficiency + Pass@1) rather than blanket 'lower cost' and general research workflows.","section":null},{"comment":"§4 and Limitations vs Appendix A: the main text states broader evaluation is limited to a small human preference study with only three participants, while Appendix A reports seven annotators × 100 tasks (700 pairwise comparisons) with Claude Opus 4.6 only, no token/cost metrics, and no non-coding agentic baselines beyond an unspecified baseline. This size inconsistency and the coding-only proxy for the dual claim are load-bearing for the 'practical research assistant' half. Please reconcile participant counts, name the baseline system, and either add non-coding efficiency metrics or explicitly demote the dual claim to coding-centric evidence plus a preference pilot.","section":null},{"comment":"§2.2 three-layer compaction and §4.2 efficiency claims: the manuscript attributes the 28.4% token reduction primarily to harness design, but free parameters (preview budgets, micro-compaction rules, ~80% semantic-compaction trigger, BM25 retrieval) are not ablated. Without at least a minimal ablation or sensitivity analysis on SWE-bench (or a subset), it is hard to attribute gains to the layered design versus other orchestration differences in Table 3 (retry/recovery, batched tools, planner-worker-critic). A compact ablation or controlled disable of one compaction layer would substantially strengthen the central efficiency claim.","section":null}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The thing worth knowing is Table 1. On full SWE-bench Verified (500 instances), ToFu beats Claude Code and OpenCode in Pass@1 across Claude Opus 4.6, GLM 5.1, and DeepSeek-v4-pro, and uses about 28% fewer tokens than Claude Code on average. That is a concrete, graded, multi-backbone result, not a toy demo.\n\nWhat is new is not the idea of an agent harness—compaction, planner-worker-critic, MCP, swarm, and coding agents already exist—but a usable open package that ships the three-layer compaction stack, cache-aware layout, BM25 memory, multilingual translate-then-reason, and a dependency-aware swarm, with MIT code and a product-ish UI (Overleaf MCP, paper reader, etc.). The design comparison to OpenCode in the appendix is useful: recovery, batched tools, and not rewriting the cached prefix look like real engineering reasons the numbers move. Multilingual MAPS:SWE-bench gains are modest but directionally sensible for English-centric models.\n\nSoft spots, in proportion. “Lower cost” in the abstract is not uniformly true in Table 1 (sometimes higher $/inst even with fewer tokens). Broader research-assistant evaluation is thin: Limitations say coding is the main focus; the human preference study is real (Appendix A: 7 annotators, 100 tasks, 77% preference) but not a substitute for multi-workflow efficiency metrics, and the main text’s “three participants” does not match the appendix. No ablations, so we do not know how much each compaction layer or the swarm buys. Free knobs (thresholds, BM25, concurrency) are engineering, not circular math.\n\nThis is for people building or studying coding agents and open harnesses who want a white-box baseline with application surface, not for someone expecting a new theory of agency. Math is not the load-bearing part; the data and citation pattern look fine for a systems paper. I would send it to peer review. Engage if you care about open agent tooling; treat the dual “research assistant for everything” claim as provisional until non-coding eval catches up.","headline":"Solid open harness paper: real SWE-bench gains and token savings vs Claude Code, dual research-assistant pitch still thinner than the coding evidence.","tokens_in":13068,"tokens_out":531,"would_cite":true,"duration_ms":6335,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"ToFu is an open, white-box agent harness that fixes more real software issues than leading coding agents while using far fewer tokens.","keywords":["agent harness","token efficiency","context compaction","SWE-bench","multilingual agents","agent orchestration","open-source agents","research assistants"],"falsifier":"Independent full runs of SWE-bench Verified with the same three backbone models showing ToFu at equal or higher average token use than the commercial baseline while matching or underperforming its Pass@1 would falsify the accuracy-plus-efficiency claim.","tokens_in":12954,"feed_emoji":"🛠️","tokens_out":909,"duration_ms":27900,"temperature":0.7,"pith_summary":"The paper argues that agent systems succeed or fail not only because of the language model, but because of the harness: the orchestration code that plans, calls tools, manages context, and recovers from errors. Existing harnesses force a trade-off between open research prototypes that are hard to use as products and polished closed products that are poor instruments for science. ToFu is offered as a single open-source system that researchers can inspect, modify, and run locally, yet still delivers application-level workflows and stronger coding results than commercial and open baselines. Its main technical bet is three-layer context compaction plus multilingual translate-then-reason and a dependency-aware swarm of sub-agents. A sympathetic reader cares because cheaper, inspectable agent loops could turn research assistants from advice-givers into systems that actually finish goals without leaking private codebases.","feed_headline":"Open agent harness beats commercial tools with 28% fewer tokens","feed_subtitle":"White-box ToFu raises coding pass rates while cutting cost, and every orchestration step stays inspectable.","key_machinery":"Three-layer context compaction: size-aware budgeting that replaces large tool outputs with previews and recoverable references; cache-aware micro-compaction that shortens cold history without rewriting cached prefixes; and query-aware semantic summarization of older turns when the context nears its limit, while keeping the current turn intact.","core_discovery":"On the full SWE-bench Verified suite of 500 instances, ToFu achieves higher Pass@1 than a leading commercial coding agent and an open coding baseline across three backbone models, while using 28.4 percent fewer tokens on average than the commercial agent (up to 43.6 percent fewer), with competitive cost and turn counts. The same harness is meant to serve dual roles: a practical research assistant that edits code, papers, and tools, and a white-box object for studying harness design itself.","pith_inferences":["If three-layer compaction generalizes past coding, the same pattern may cut cost for long-horizon research agents that browse papers, edit manuscripts, and run experiments overnight.","Future harness comparisons may need to report recovery under stream failures and cache-hit stability as first-class metrics alongside Pass@1.","Translate-then-reason may need language-specific safeguards when technical terms or code-adjacent prose are mistranslated.","Open harnesses with structured tool-error return and automatic retry may matter more for unattended work than small gains on curated benchmarks alone."],"forward_implications":["Privacy-sensitive researchers can run capable agent loops locally under an open license instead of shipping codebases to closed services.","Harness-evolution research gains a baseline that is both strong on coding benchmarks and usable as a product surface.","Non-English users can keep their interface language while the agent reasons in English, improving multilingual coding outcomes on average.","Longer test-time token budgets are not automatically better; orchestration quality can beat simply spending more tokens.","Protocol-based tool adapters can turn manuscript, version-control, and note systems into first-class agent environments."],"fun_headline_variants":["White-box ToFu tops commercial agents with 28% fewer tokens","ToFu raises SWE-bench Pass@1 while cutting tokens 28%","Open inspectable harness beats closed agents on coding tasks","ToFu: higher pass rates and 28% token savings vs commercial","White-box research harness trims tokens 28% with better Pass@1"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That strong coding-benchmark results plus a small human preference study are enough to support the dual claim of a general research assistant and a reusable harness-research baseline.","fun_headline_variants_meta":{"raw":{"variants":["White-box ToFu tops commercial agents with 28% fewer tokens","ToFu raises SWE-bench Pass@1 while cutting tokens 28%","Open inspectable harness beats closed agents on coding tasks","ToFu: higher pass rates and 28% token savings vs commercial","White-box research harness trims tokens 28% with better Pass@1"]},"model":"grok-4.5","effort":"low","cost_usd":0.003618,"raw_usage":{"total_tokens":1157,"prompt_tokens":735,"num_sources_used":0,"completion_tokens":78,"cost_in_usd_ticks":36180000,"prompt_tokens_details":{"text_tokens":735,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":344,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":735,"tokens_out":78,"duration_ms":3266,"temperature":1.0,"reasoning_tokens":344,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T05:40:55.404386+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Independent full runs of SWE-bench Verified with the same three backbone models showing ToFu at equal or higher average token use than the commercial baseline while matching or underperforming its Pass@1 would falsify the accuracy-plus-efficiency claim.","supporting_citations":[],"review_version":1}