{"id":"4a0c94c4-72ab-4a0c-9332-78e08ce008cd","arxiv_id":"2607.11149","paper_version":2,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Storage footprint is a measurable, independent resource axis for LLM agents: equally accurate configurations differ 15.7× in retained bytes and most of the spread is removable duplication.","lead":"This paper introduces AgentFootprint, a benchmark that measures how much disk space an LLM agent run leaves behind—logs, checkpoints, traces, and hidden state. It finds that agent setups with identical task accuracy differ by up to 15.7× in retained bytes, and that a content-addressed store can remove most of that overhead while preserving the ability to reconstruct the run's history.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Full-filesystem boundary not verified; out-of-boundary retention could shift headline spreads.","rationale":"The reader's weakest_assumption was exactly the sandbox boundary: 'If significant persistence occurs outside the sandbox boundary (e.g., system temp, remote tracing/telemetry), the reported Stotal and D values and the 15.7× and 6.7× spreads would shift.' I agree this is the most load-bearing unverified condition. The paper is transparent about the limitation in Appendix N and provides an internal audit of workspace/home deltas, plus a Docker cross-check for LangGraph only. But the central claim—that storage footprint is a meaningful, consistently measurable resource dimension—depends on Stotal being complete and attribution-sound. If the full-filesystem audit reveals out-of-boundary persistence that is framework-dependent, then every ratio-based headline (15.7×, 6.7×, 4.8–32.7×, and the α growth classes) is at risk, because those ratios are computed from Stotal. This is not an accusation of error; it is a concrete, testable gap that the authors themselves flag. The right verdict is therefore conditional acceptance: the benchmark's methodology and reported measurements are credible and internally consistent, but the central quantitative claims should be considered provisional until the full-filesystem boundary test is run. If the test passes, ACCEPT stands with no further changes. I am not lowering confidence on any other ground: the fixed-trace control, backend replications, calibration studies, and CAS restoration checks are strong independent support, and the acknowledged limitations about α being descriptive, wild data being exports, and InfiAgent's self-benchmarking status do not by themselves undermine the core metric proposal.","tokens_in":23443,"tokens_out":8774,"duration_ms":84028,"concrete_test":"Run the fixed-trace control and one full file-QA repetition for all seven persisting frameworks in Docker containers with a full rootfs diff—including /tmp, /var/tmp, /run, and any XDG dirs not already redirected—and with network egress blocked or logged. Compare container-derived Stotal and per-framework rankings against the sandbox numbers in Table 1. Settling thresholds: if any framework's full-container Stotal exceeds the sandbox value by more than the 11% repetition-level standard deviation, or if the 15.7× spread changes by more than 10%, the boundary assumption fails and all headline spreads require recomputation; otherwise the concern is retired.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"AgentFootprint's central numbers all reduce to Stotal, defined in §3.1 as bytes present after the run, attributable to it, absent before it, measured by diffing workspace and HOME/XDG snapshots. Appendix N explicitly concedes that 'the redirected boundary does not cover system temp directories or remote telemetry; a full-filesystem container audit is the extension path for adversarial submissions.' The only container check (§4.3) covers LangGraph on three tasks and reports 0–8.5% deviation from sandbox measurements; it does not establish that the other frameworks—or LangGraph on the growth stress task—leave no attributable bytes in /tmp, /var/tmp, SQLite journal directories, or unredirected XDG state. The headline 15.7× equal-accuracy spread, the 6.7× fixed-trace spread, the superlinear α growth, and the CAS reduction factors are all ratios of Stotal. If a framework-dependent out-of-boundary component exists, even at a few percent of Stotal, the spreads and cross-framework rankings could shift. The paper's internal size-inventory audit validates the workspace/home delta but is silent on paths outside that boundary. This is not an internal inconsistency—it is an explicitly acknowledged, empirically unverified boundary assumption that the benchmark's own text identifies as unfinished.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes AgentFootprint, a benchmark and metric suite for measuring persistent storage left by LLM agent runs. The six metrics (Stotal, composition, D, α, C, R) are applied in 1,061 sandboxed runs across eight agent frameworks. The controlled study reports a 15.7× spread in retained bytes among 100%-accuracy default configurations, a 6.7× spread under fixed-trace replay, superlinear retention growth for full-history frameworks, zero retention for an in-memory framework, and a 1,617× spread in exported trajectory volume across 108 SWE-bench Verified submissions with no detectable resolve-rate correlation. A content-addressed store reduces retention 4.8–32.7× while preserving reconstructability scores. The paper includes extensive reproducibility artifacts, calibration studies, and sensitivity analyses.","tokens_in":23642,"tokens_out":8894,"duration_ms":86248,"significance":"If the measurements hold, this paper makes a strong case for reporting storage as a resource axis in agent evaluation. Its main methodological contribution is serialization-aware measurement: logical-stream chunking recovers D=12.1 where raw-file chunking sees 1.01, and calibration on synthetic stores confirms the meter on known ground truth. The fixed-trace control cleanly isolates persistence-layer amplification from agent behavior. The empirical effort is unusually careful: exhaustive per-run accounting, a bootstrap CI for the headline spread, two backend replications with rank correlations 0.96/0.89, threshold-sensitivity audits, and an automated verification suite in the artifact. The central claim is credible within the declared sandbox boundary; the open question is whether that boundary fully captures the storage phenomenon the title promises.","major_comments":[{"comment":"The headline numbers all reduce to Stotal, but Stotal is measured only from workspace/HOME snapshots. Appendix N explicitly concedes that the redirected boundary does not cover system temp directories or remote telemetry. The only container validation (§4.3) is LangGraph on three tasks (0–8.5% deviation). For the other seven frameworks, and for LangGraph on the growth stress task, there is no check for writes to /tmp, /var/tmp, SQLite journal directories, or unredirected XDG state. If any framework leaves a few percent of its retention outside the boundary, the 15.7×, 6.7×, and 3.9× spreads and the CAS factors could shift. Because the paper identifies this as an 'extension path,' it is an acknowledged limitation, but it is load-bearing for the central claims. Please extend the container audit to all eight adapters on a representative task set, or provide explicit evidence that none of th","section":"§3.2 / Appendix N"}],"minor_comments":[{"comment":"The statement that OpenAI Agents' hosted tracing is disabled because it is remote retention should be moved to §3.1 next to the boundary definition; otherwise readers may not realize Stotal is local-only.","section":"Appendix B / §3.1"},{"comment":"InfiAgent is the authors' own prior framework; while it is cited, an explicit sentence in the benchmark description would remove any appearance of conflict.","section":"§4.2 / references"},{"comment":"The caption defines α as a growth exponent but does not state that the value for non-full-history frameworks is a single-run fit while the full-history trio has three seeds; this distinction appears only in the text.","section":"Table 1 caption"}],"recommendation":"major_revision","confidential_remarks":"The sandbox-boundary issue is the only substantive blocker. I would accept after the additional audit is performed or clearly scoped away. Also worth flagging to the editor: the authors measure their own InfiAgent framework; a disclosure in the main text would be appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Storage footprint is a genuinely new and underexplored axis for LLM agent evaluation, and this paper is the first to measure it systematically across frameworks. The headline findings—15.7x spread at equal accuracy, 6.7x persistence-layer amplification, superlinear growth for full-history designs—are backed by explicit run counts, repetition seeds, bootstrap CIs, and calibrations on synthetic stores. That is more than most benchmark papers do, and the serialization-aware metric suite is the real contribution: the D=1.01 vs 12.1 result on the same store is a measurement trap most people would have missed. The fixed-trace control is a strong design because it isolates the persistence layer from agent behavior. The CAS demonstration is honestly framed as an existence proof, not a new storage system. The paper is unusually transparent: alpha is descriptive, results are version-specific, wild data are exports, and the sandbox boundary is explicitly scoped.\n\nThe main soft spot is that boundary. All headline numbers are ratios of Stotal, which is measured as the workspace/home delta. App N admits the boundary excludes system temp and remote telemetry, and the container check only covers LangGraph on three tasks. So out-of-boundary retention is not empirically ruled out for the other frameworks. That could shift the spreads, but it would need to be both large and framework-dependent to change the conclusions. Given the magnitudes, I read this as a disclosed limitation rather than a hidden flaw. Still, a careful referee should ask for a broader container audit or a stronger argument that outside bytes are negligible.\n\nThe InfiAgent point is minor: it is the authors' prior work, cited but not flagged as self-measured in the main text. The artifact lacking a commit hash in the text is annoying but not substantive.\n\nThis paper deserves a serious referee. I would send it out, expecting acceptance after some boundary validation or clearer caveats. It is also a good reading-group candidate: the methodology makes you think about what we choose to measure in agent evaluation.","headline":"Storage footprint is a real, measurable evaluation axis; the disclosed boundary caveat is a soft spot, not a fatal one.","tokens_in":24235,"tokens_out":2525,"would_cite":true,"duration_ms":24047,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM agent runs leave a persistent disk footprint that is as informative as accuracy and inference cost, and it spans 15.7x across equal-accuracy configurations once measured at the logical-stream level.","keywords":["LLM agents","storage footprint","persistent state","benchmarking","duplication","serialization","reconstructability","content addressing"],"falsifier":"Run the same eight configurations with a full-filesystem container audit covering system temp and remote endpoints; if a configuration's per-task retained bytes outside the declared boundary exceed the reported delta by a meaningful fraction, the metric's attribution boundary fails and the headline spreads need revision.","tokens_in":23237,"feed_emoji":"💾","tokens_out":4265,"duration_ms":38142,"temperature":0.7,"pith_summary":"This paper argues that the bytes an LLM agent run leaves on disk—logs, checkpoints, snapshots, traces—are a resource dimension as worth reporting as accuracy and inference cost, and it builds a benchmark, AgentFootprint, to measure them. The benchmark's serialization-aware meter reads logical content streams rather than raw files, so duplication hidden by database paging and JSON escaping becomes visible. Across eight representative frameworks on identical tasks, the paper finds documented configurations with perfect accuracy retain between 0.32 and 5.10 MB per task—a 15.7x spread—and that replaying one fixed trajectory through different persistence layers alone produces a 6.7x spread. Full-history checkpointing grows superlinearly with repeated observations, while a content-addressed store cuts retention 4.8-32.7x without changing any reconstructability score. The upshot: storage can be reported as a first-class metric, and exact history reconstruction does not require megabytes.","feed_headline":"Agent disk residue spans 15.7x at equal accuracy","feed_subtitle":"A new benchmark measures the persistent bytes agents leave behind—and shows exact history reconstruction needs far fewer.","key_machinery":"The central object is the six-metric storage-footprint suite (total retention, channel composition, duplication factor D, growth exponent alpha, compressibility C, and 0-3 reconstructability score R), built on a per-run sandbox delta and a serialization-aware meter. The meter extracts logical content streams—SQLite cells and JSONL records—before fingerprinting, so duplication masked by page fragmentation and JSON escaping becomes visible, and content probes estimate how many times input content is stored. A fixed-trace control replays an identical trajectory through each framework's persistence adapter to isolate persistence-layer amplification from agent behavior.","core_discovery":"The central claim is that persistent storage footprint is a meaningful, orthogonal resource axis for LLM-agent evaluation, and that it has been systematically under-measured because serialization masks duplication. Measuring SQLite cells and JSONL records as logical streams instead of raw bytes raises a naive duplication factor from 1.01 to 12.1 on one store, and content probes confirm the repetition. With this measurement, equal-accuracy configurations differ by 15.7x in retained bytes, the persistence layer alone accounts for a 6.7x spread on identical content, full-history frameworks grow with exponent up to 1.95 on a repeated-observation task, and a content-addressed store reduces retent","pith_inferences":["If storage becomes a standard reporting axis, frameworks that bundle recovery, audit, and workflow-state capabilities will need capability-normalized efficiency metrics, not just raw byte counts, to be compared fairly.","The serialization-masking effect generalizes beyond agents: any system persisting structured data via SQLite or JSON is vulnerable to order-of-magnitude duplication that raw byte measurement misses.","A testable extension: run the same metric suite on long-horizon production workloads with different archival policies to see whether the superlinear growth exponent predicts actual retention obligations at fleet scale.","The content-addressed reference store suggests delta encoding, not just exact-match deduplication, could close the remaining gap between opaque compression and schema-preserving stores."],"forward_implications":["Benchmark authors can report retained bytes, duplication, growth, compressibility, and reconstructability alongside accuracy and inference cost.","Equal-accuracy configurations can differ by 15.7x in retained bytes, so a success-only leaderboard hides a large operational difference.","Full-history persistence grows superlinearly on repeated observations (up to a 1.95 exponent), implying long-running agents need archival policies or windowed context to bound disk usage.","Content addressing removes 4.8-32.7x of retention while preserving exact conversation-history reconstruction, showing most retained bytes are removable redundancy.","Exported trajectory volume in the wild shows no detectable correlation with task resolution, so storage volume is not a proxy for quality."],"fun_headline_variants":["Storage: the untracked cost of agent runs","Equal accuracy, 15.7x disk spread","Agent disk waste: up to 32.7x reducible","Measure agent storage, not just accuracy","Serialization masks 12x duplication in agent data"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the fresh-sandbox diff captures every byte the run leaves behind; if significant persistence lands outside the workspace and home boundary (system temp, remote telemetry), the reported totals and spreads shift.","fun_headline_variants_meta":{"raw":{"variants":["Storage: the untracked cost of agent runs","Equal accuracy, 15.7x disk spread","Agent disk waste: up to 32.7x reducible","Measure agent storage, not just accuracy","Serialization masks 12x duplication in agent data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001128,"raw_usage":{"total_tokens":4542,"prompt_tokens":774,"completion_tokens":3768,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":3691}},"tokens_in":518,"tokens_out":3768,"duration_ms":25558,"temperature":1.0,"reasoning_tokens":3691,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T06:58:38.136622+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same eight configurations with a full-filesystem container audit covering system temp and remote endpoints; if a configuration's per-task retained bytes outside the declared boundary exceed the reported delta by a meaningful fraction, the metric's attribution boundary fails and the headline spreads need revision.","supporting_citations":[],"review_version":2}