{"id":"04c866a9-c6a1-4c22-958b-de56d2c178a8","arxiv_id":"2604.19572","paper_version":3,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"TACO is a self-evolving, plug-and-play compression framework that filters low-value terminal observations while preserving task-relevant signals, yielding 1-4% accuracy gains and better token efficiency on TerminalBench and related benchmarks.","lead":"The paper presents TACO, a training-free framework that learns to compress noisy terminal outputs for AI agents by discovering rules from past interactions. Smart generalists might read it to understand practical ways to make long-running software agents cheaper and more reliable without retraining models.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Discovered rules from limited trajectories may not generalize reliably to heterogeneous terminal environments without discarding critical signals.","rationale":"This directly matches the reader's weakest assumption on generalization from limited trajectories. The concern is internal to the argument rather than external consensus; small reported gains make the robustness question especially load-bearing. Full text would allow checking trajectory count and diversity, but the risk remains even if methods are sound.","tokens_in":1806,"tokens_out":314,"duration_ms":33674,"concrete_test":"Apply the rules evolved from the original TerminalBench trajectories to a held-out terminal task using a new command set and output format (e.g., a different OS or language-specific tool not present in training trajectories); measure whether accuracy drops below the no-compression baseline or whether task-critical observations are filtered.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that rules automatically extracted and refined from interaction trajectories enable workflow-adaptive filtering that preserves task-relevant observations across highly variable terminal outputs (different repos, commands, error formats, and execution states). The abstract positions this as the key advantage over fixed heuristics. However, with only finite trajectories available for rule discovery, the process risks learning brittle patterns that over-compress in unseen settings—for instance, a rule discarding repetitive log lines might eliminate unique stack-trace details needed for debugging in a new environment. The reported 1-4% gains on TerminalBench and related benchmarks do not isolate whether this generalization holds or whether performance relies on benchmark similarity to the source trajectories.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript proposes TACO, a plug-and-play, training-free, self-evolving framework for terminal agents. It automatically discovers, refines, and reuses structured compression rules from interaction trajectories to enable workflow-adaptive filtering of low-value terminal outputs while preserving task-relevant observations. Experiments on TerminalBench (TB 1.0 and 2.0) plus SWE-Bench Lite, CompileBench, DevEval, and CRUST-Bench report consistent 1-4% accuracy gains across agent scaffolds and backbones, plus token reductions under fixed budgets, with code released publicly.","tokens_in":1936,"tokens_out":452,"duration_ms":35877,"significance":"If the generalization claim holds, the work offers a practical route to scaling long-horizon terminal agents without training or brittle fixed heuristics, directly addressing context saturation in heterogeneous environments. Public code availability strengthens reproducibility.","major_comments":[{"comment":"Experiments section: the reported 1-4% gains on TerminalBench do not isolate whether performance stems from adaptive rule discovery or from similarity between the finite trajectories used for rule extraction and the benchmark environments. No out-of-distribution terminal settings (different repos, error formats, or execution states) are evaluated to test the central claim that discovered rules preserve critical signals without over-compression.","section":"Experiments"},{"comment":"Methods / Evaluation details: insufficient information is given on trajectory selection for rule discovery, data exclusion criteria, and statistical tests (e.g., variance across runs or significance of the 1-4% deltas). These omissions make it impossible to assess whether the self-evolving component is load-bearing or whether gains are robust.","section":"Methods"}],"minor_comments":[{"comment":"Abstract: the phrase 'around 2%-3% under the same token budget' should specify the exact benchmark and condition.","section":"Abstract"},{"comment":"Notation: clarify how 'structured compression rules' are represented and updated across iterations; the current description is high-level.","section":"Framework Description"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"Thank you for the opportunity to respond to the referee's comments. We address the major comments point by point below, providing clarifications and outlining planned revisions to strengthen the manuscript.","responses":[{"response":"We thank the referee for highlighting this important point regarding the isolation of the self-evolving component's contribution. Our experiments demonstrate consistent improvements across a range of benchmarks that feature diverse terminal environments, repositories, and execution states, which provides evidence for the generalizability of the discovered rules. The self-evolving aspect is central as rules are refined from interaction trajectories within each workflow. To more rigorously address potential distribution similarity concerns, we will include additional out-of-distribution evaluations in the revised manuscript, such as testing on unseen repositories and error formats.","revision_made":"yes","referee_comment":"[Experiments] Experiments section: the reported 1-4% gains on TerminalBench do not isolate whether performance stems from adaptive rule discovery or from similarity between the finite trajectories used for rule extraction and the benchmark environments. No out-of-distribution terminal settings (different repos, error formats, or execution states) are evaluated to test the central claim that discovered rules preserve critical signals without over-compression."},{"response":"We agree that the manuscript would benefit from more detailed descriptions in these areas. In the revised version, we will expand the Methods and Evaluation sections to include specifics on trajectory selection criteria for rule discovery, data exclusion rules applied during the process, and comprehensive statistical analyses including variance across multiple runs and significance testing for the performance deltas.","revision_made":"yes","referee_comment":"[Methods] Methods / Evaluation details: insufficient information is given on trajectory selection for rule discovery, data exclusion criteria, and statistical tests (e.g., variance across runs or significance of the 1-4% deltas). These omissions make it impossible to assess whether the self-evolving component is load-bearing or whether gains are robust."}],"tokens_in":1439,"tokens_out":414,"duration_ms":47514,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point with this TACO paper is that it lets terminal agents learn their own compression rules from interaction histories to handle noisy outputs better, leading to small accuracy bumps and token savings on the benchmarks they tried.","headline":"TACO provides a workable self-evolving compression method for terminal agents with small benchmark gains, though generalization across environments remains a key uncertainty.","tokens_in":2434,"tokens_out":114,"would_cite":false,"duration_ms":38253,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"Practical self-evolving rule compression for terminal agents; no overlap with RS forcing chain or J-cost structures","alignment":"orthogonal","rationale":"The paper's machinery (Global Rule Pool evolution, intra-task rule refinement from trajectories, retention-based convergence, conservative filtering preserving critical signals) is standard agent-engineering practice for context management. It neither invokes nor parallels any RS theorem: no J(x) = ½(x + x⁻¹) − 1, no φ-ladder, no 8-tick periodicity, no absolute-floor distinguishability, and no parameter-free derivation of constants. The domain (cs.CL terminal-agent benchmarks) lies outside RS scope.","tokens_in":61419,"confidence":"high","tokens_out":154,"duration_ms":9149,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Terminal agents can self-discover compression rules from their interaction histories to filter noise while keeping task-critical signals.","keywords":["terminal agents","context compression","self-evolving rules","observation filtering","agent efficiency","long-horizon workflows","terminal benchmarks","trajectory-based adaptation"],"falsifier":"A controlled test on a new terminal benchmark where applying the learned rules causes agents to overlook key error messages or status updates and achieve lower success rates than the uncompressed baseline would disprove the central claim.","tokens_in":2708,"feed_emoji":"💻","tokens_out":661,"duration_ms":45008,"temperature":0.7,"pith_summary":"The paper presents TACO as a training-free framework that lets terminal agents automatically discover, refine, and reuse structured rules for compressing their observation histories. This targets the buildup of noisy terminal outputs in long workflows, where keeping everything leads to context overload and discarding too much risks losing needed feedback. Because terminal setups vary widely across commands and states, fixed compression approaches fall short, but learning rules from actual trajectories allows adaptation to each workflow. Tests on TerminalBench and related benchmarks show consistent gains in task success alongside lower token use across different agent models and scaffolds. The approach demonstrates that evolving compression from experience offers a practical route to more reliable long-horizon terminal agents.","feed_headline":"Self-evolving rules compress terminal agent histories","feed_subtitle":"Learning from past trajectories filters noise, cuts token use, and raises accuracy 1-4 percent on TerminalBench tasks.","key_machinery":"The self-evolving discovery of structured compression rules from interaction trajectories, which refines filtering to match specific workflow needs without external training.","core_discovery":"TACO automatically discovers, refines, and reuses structured compression rules from interaction trajectories, enabling workflow-adaptive filtering of low-value terminal outputs while preserving task-relevant observations and yielding accuracy gains of 1 to 4 percent on TerminalBench along with token reductions on additional benchmarks such as SWE-Bench Lite.","pith_inferences":["The rule-evolution process could be applied to other long-horizon agent settings that accumulate noisy observations, such as web or robotic environments.","Monitoring how the discovered rules change over many more trajectories might reveal whether compression quality continues to improve with extended use.","Pairing the framework with dynamic context-window managers could produce additive efficiency benefits on even longer tasks."],"forward_implications":["Agents equipped with TACO record 1 to 4 percent higher accuracy on TerminalBench across multiple strong models.","Under fixed token budgets the method delivers roughly 2 to 3 percent accuracy gains by removing low-value outputs.","The same compression approach reduces total token consumption on SWE-Bench Lite, CompileBench, DevEval, and CRUST-Bench while holding or raising task success rates.","TACO integrates as a plug-and-play module into existing agent scaffolds without requiring model retraining or changes to the backbone."],"fun_headline_variants":["TACO evolves compression rules for efficient terminal agents","Structured rules filter terminal outputs via self-evolution","Terminal agents gain accuracy from self-evolving compression rules","Self-discovered rules enable efficient long-horizon terminal workflows"],"cache_read_input_tokens":64,"weakest_assumption_plain":"Rules discovered from limited interaction trajectories will generalize across varied terminal environments without removing signals essential for correct task completion.","fun_headline_variants_meta":{"raw":{"variants":["TACO evolves compression rules for efficient terminal agents","Structured rules filter terminal outputs via self-evolution","Terminal agents gain accuracy from self-evolving compression rules","Self-discovered rules enable efficient long-horizon terminal workflows"]},"model":"grok-4.3","cost_usd":0.00625,"raw_usage":{"total_tokens":2888,"prompt_tokens":722,"num_sources_used":0,"completion_tokens":59,"cost_in_usd_ticks":62503000,"prompt_tokens_details":{"text_tokens":722,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2107,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":722,"tokens_out":59,"duration_ms":25050,"temperature":1.0,"reasoning_tokens":2107,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-19T17:40:00.143407+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A controlled test on a new terminal benchmark where applying the learned rules causes agents to overlook key error messages or status updates and achieve lower success rates than the uncompressed baseline would disprove the central claim.","supporting_citations":[],"review_version":2}