{"id":"550ddb20-dbc5-4e33-98d3-1856c95ffcb2","arxiv_id":"2608.04830","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"ContextWeave turns real multi-month human work histories into executable agent tasks and shows that memory of prior tasks improves downstream task outcomes across all five tested base models.","lead":"The paper builds ContextWeave, a benchmark from the multi-month office work of 14 real people, converted into 1,005 executable AI tasks. It reports that giving agents memory of earlier tasks improves later performance across all tested models, with richer prior experience helping more than compact summaries.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline memory gains rest on GPT-5.5 rubric scores, and the grader is explicitly anchored to an LLM-reconstructed reference that recalled history is engineered to resemble; without human-calibrated validation the central claim is unestablished.","rationale":"ContextWeave is a substantial contribution: executable Docker tasks, temporal-relevance statistics, six memory components, five base models, and downstream-oriented diagnostics. The design is largely sound, and the paper is unusually candid in Section 5.2.3 about the summary-versus-in-context comparison being an association, and in the Limitations section about ongoing rubric calibration. I found no formal error in the construction; the weakest point is the evaluative loop. Section 4.3 reconstructs tasks with LLM agents and verifiers; Section 4.4 rewrites trajectories to match the reference diff; Section B.7.1 grades candidate workspaces against that same reference as an 80-point anchor. This creates a systematic prior in favor of with-recall runs, which are more likely to reproduce the reference's style and file conventions. The paper's own caveats, human validation only coarse-grained and sampled, with no per-rubric agreement, mark exactly where the central claim is unsupported. Secondary issues, such as the repeated MemoryBench naming in the Limitations and appendix prompts and the absence of a commit hash or artifact manifest, should be fixed but do not change the verdict. If the proposed human and reference-blind grading checks reproduce the reported deltas, the central claim would be considerably strengthened; if not, the unconditional version of the claim should be withdrawn. The reader's conditional verdict already reflects this uncertainty, so no verdict change is needed.","tokens_in":39894,"tokens_out":10134,"duration_ms":115827,"concrete_test":"Take a stratified random sample of 100-150 core tasks spanning all participants and the six memory components. (1) Have independent human raters, blind to condition, score final workspaces with the same minimum/reference/better criteria and choose between anonymized with-recall and without-recall outputs; compute human-vs-GPT-5.5 agreement on the sign and size of the delta. (2) Re-run Workspace grading on the same sample with criteria-only input, omitting the reference image, to see whether the memory gain persists when the grader cannot anchor to the reference. If the human delta or the criteria-only delta is substantially smaller than the reported 68-to-78 and 41-to-70 gains, the central claim is a grading artifact rather than a memory effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All reported downstream outcomes are LLM-judged. Section 4.5 defines Workspace and Preference Scores via GPT-5.5, and Sections 5.2.1 and 5.3.1 state that every grading, annotation, and judgment call uses GPT-5.5. The load-bearing risk is not generic judge noise: the Workspace grader in Section B.7.1 is instructed to treat the reference Docker image as the roughly 80-point anchor and to derive task requirements from it, while the with-recall entries are built from trajectories that Sections 4.3-4.4 reconstruct and rewrite backwards from the reference document diff. The with-recall condition is therefore aligned to the same LLM-generated reference that the grader compares against. The headline gains (68.08 to 78.20 Workspace, 41.50 to 70.60 Preference) could partly measure self-consistency, an LLM judge rewarding outputs that imitate an LLM-generated reference, rather than quality that transfers to human users. Section 6 concedes rubric calibration is ongoing and human validation is only coarse-grained and sampled, with no per-task or per-rubric agreement and no condition-level human preference data. Since every delta in Tables 1 and 2 is a function of these scores, this is the load-bearing premise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents ContextWeave, a benchmark that reconstructs multi-month, privacy-anonymized office workflows of 14 participants into 1,005 executable Docker tasks (568 core evaluation tasks). For each target task, an agent is run with or without access to prior task history, and downstream outcomes are scored by GPT-5.5 using a Workspace Score and a Preference Score, alongside four diagnostics (relevance, continuity, solvability, hallucination robustness). Experiments with six memory components under a fixed execution model and with five base models under mem0 report that recall improves both outcome scores for every tested model, with the strongest configuration raising the Workspace Score from 68.08 to 78.20 and the Preference Score from 41.50 to 70.60. The paper further argues that in-context experience memory outperforms compact summary memory while being more susceptible to misleading recall.","tokens_in":40017,"tokens_out":7212,"duration_ms":79098,"significance":"If the reported effects survive validation, ContextWeave is a valuable community resource: it provides real temporal dependencies, executable environments, a controlled paired protocol, and multi-faceted diagnostics, and the public repository is a clear strength. The no-recall baseline anchors the comparison, and the fact that conditions differ only by injected context is a sound experimental skeleton. The main risk is that every headline number is an LLM-judged point estimate with limited human calibration, so the benchmark's central quantitative claims are not yet established.","major_comments":[{"comment":"All downstream scores are GPT-5.5 rubric outputs, and the Workspace grader is explicitly instructed to treat the reference image as 'roughly around the 80-point level' and to derive task requirements from it. The Limitations section concedes that rubric calibration is ongoing and human validation is only a coarse-grained sampled subset. Since every delta in Tables 1 and 2 is computed from these scores, the central claim that memory improves downstream performance needs per-rubric human-model agreement (e.g., Cohen's kappa or ICC), bootstrap confidence intervals for the headline scores, and a human pairwise preference validation on a random sample of with-recall vs. without-recall trajectories.","section":"Section 4.5, 5.2.1, 6, B.7.1"},{"comment":"With-recall conditions consume trajectories that were reconstructed and rewritten backward from the reference document diff via 'result-aligned trajectory' rewriting, while the Workspace grader derives its requirements from the same reference-derived workspace. This is not a definitional circularity because the no-recall condition anchors the comparison, but it means the measured gain could partly reflect self-consistency between an LLM judge and an LLM-generated reference rather than quality that transfers to human users. Please test this by (i) rerunning grading with a different judge model and (ii) comparing model-based deltas to human expert ratings on a held-out sample.","section":"Sections 4.3-4.4 vs. B.7.1"},{"comment":"No uncertainty quantification is reported. The tables give single point estimates with no per-task variance, confidence intervals, or paired significance tests, so the claim that recall improves every tested base model cannot be assessed; for example, the Workspace deltas in Table 2 range from +2.19 to +5.61 and may be within noise. Please report bootstrap confidence intervals and paired tests across the 568 core tasks for all downstream deltas.","section":"Tables 1 and 2"},{"comment":"The claim that in-context experience memory is more effective than compact summaries is confounded because the memory components differ in retrieval strategy and context length, as the paper itself acknowledges only in the last sentence of Section 5.2.3. The abstract and Section 6 present this comparison as a finding. Please either soften the wording or add a controlled comparison at a fixed token budget.","section":"Section 5.2.3 and abstract"},{"comment":"The benchmark's realism claim rests on the unexamined assumption that result-aligned trajectory rewriting preserves the realistic memory signal of prior experience. Because all memory components consume these rewritten trajectories, this assumption is load-bearing for external validity. Please provide evidence that rewritten trajectories remain natural and representative, for example by measuring divergence between the original and rewritten trajectories and by human reading of a sample.","section":"Section 4.4"}],"minor_comments":[{"comment":"The Limitations section repeatedly refers to 'MemoryBench' and the appendix prompts use 'Membench' instead of 'ContextWeave'; please make the naming consistent throughout.","section":"Section 6 and Appendix B"},{"comment":"The caption says the temporal panels use 1,007 task IDs, while the paper reports 1,005 total tasks and 568 core tasks; please reconcile these counts.","section":"Figure 2 caption"},{"comment":"The Preference Score normalization is described only as 'weighted rubric scores normalized to 0-100'; please specify the weighting and normalization formula, since this metric drives a headline claim.","section":"Section 4.5"},{"comment":"The Solvability metric is reported as 'Problem Solvability Rate' in Table 1 and as 'Recall-Solvable Problems' in Table 2; the relationship between these two quantities should be stated explicitly.","section":"Tables 1 and 2"},{"comment":"The cosine-similarity threshold of 0.65 for relevance diagnostics is presented as a fixed choice; please justify it and report sensitivity to the threshold.","section":"Section 5.2.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a large benchmark release and the underlying resource may be valuable, but the copy-and-paste remnants from a 'MemoryBench' template in the Limitations section are a quality-control concern. The decisive issue is the absence of human-calibrated validation for the LLM-judged outcome metrics; if the authors can supply per-rubric agreement and uncertainty estimates, the paper would be within reach."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is the first benchmark I have seen that turns real multi-month human document workflows into executable, stateful agent tasks with downstream outcome metrics rather than QA-over-history. That is a real contribution, and it is new relative to MemoryArena's constructed interdependencies and the RULER/LoCoMo/LongMemEval QA-style line. The controlled trajectory alignment, the no-recall baseline, the participant-specific preference rubrics, and the sweep across six memory components and five base models give the paper a solid empirical skeleton. If you work on agent memory, you should read it.\n\nThe soft spots are concentrated in one place: every headline number is a GPT-5.5 rubric judgment, and the rubric is anchored to an LLM-reconstructed reference that the with-recall trajectories were deliberately aligned to. Section B.7.1 tells the grader to treat the reference Docker image as roughly the 80-point anchor and to derive task requirements from it; Sections 4.3-4.4 reconstruct instructions and rewrite trajectories backwards from the same document diff. So a non-trivial part of the 68.08-to-78.20 and 41.50-to-70.60 gains could be self-consistency: an LLM judge rewarding outputs that imitate an LLM-generated reference. The paper itself concedes in Section 6 that rubric calibration is ongoing and human validation is only coarse-grained and sampled. That makes the precise magnitudes provisional, not the directional finding.\n\nI do not think this flaw is fatal. The paired with- versus without-recall design is controlled, the no-recall baseline is sensible, and the claim that memory helps every tested model is credible even with grader noise. The abstract does overstate the summary-versus-in-context comparison; Section 5.2.3 calls it an association, and the abstract should too. The repeated 'MemoryBench' naming in the Limitations and appendix prompts is a minor but real artifact-release issue, and reproducibility is currently unverified: there is a GitHub link but no commit hash, license, manifest, or dataset location.\n\nWho gets value: anyone building or evaluating memory systems for agents. The benchmark artifact is the contribution; the effect sizes should be treated as preliminary until human-calibrated validation and error bars appear. I would send this to peer review. A serious referee should push on the grader-realignment loop and on artifact completeness, but desk rejection would be a mistake.","headline":"A genuinely useful longitudinal agent-memory benchmark whose headline numbers are all LLM-judged; the benchmark is worth serious review, but the central effect sizes need human-calibrated validation before they are quoted.","tokens_in":40765,"tokens_out":1289,"would_cite":true,"duration_ms":18642,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Recalling prior work improves language agents' performance on real office workflows: the strongest memory component raises Workspace Score from 68.08 to 78.20 and Preference Score from 41.50 to 70.60.","keywords":["agent memory","workflow benchmark","longitudinal evaluation","in-context experience","preference alignment","workspace quality","memory diagnostics","LLM agents"],"falsifier":"Take a random sample of, say, 100 of the 568 core tasks, run the no-recall and strongest with-recall conditions, and have human annotators who do not know which output has memory blindly rate workspace usefulness and preference alignment; if human-rated gains are substantially smaller than the GPT-5.5-rated gains (68.08 to 78.20 and 41.50 to 70.60), the central claim that memory improves downstream performance in real workflows would be called into question.","tokens_in":71,"feed_emoji":"🧠","tokens_out":6142,"duration_ms":103532,"temperature":0.7,"pith_summary":"ContextWeave is a benchmark that tests whether language agents perform better on realistic office tasks when they can recall their past work. Built from the multi-month, privacy-anonymized workflows of 14 people, it turns 1,005 real document-editing sessions into 568 executable evaluation tasks with Docker environments and task-specific rubrics. The paper's central claim is that access to prior experience improves downstream task performance, with every one of five base models benefiting from recall in both workspace quality and preference alignment. It further argues that concrete in-context experience helps agents continue workflows better than compact summaries, while carrying a higher risk of misleading recall. A sympathetic reader would care because most existing memory benchmarks only test retrieval or question-answering, not whether memory actually makes an agent's final work better.","feed_headline":"Memory recall lifts AI agent scores in realistic workflow benchmark","feed_subtitle":"Past experience improved workspace quality and preference alignment for every tested base model.","key_machinery":"The central object is the sequential task stream $D=(T_1,\\dots,T_n)$ with the memory-induced gain $\\Delta R_M(T_i)=R_M(T_i)-R(T_i)$, measured as the difference between an agent's performance on task $T_i$ with and without access to its preceding task histories. The operative machinery is the benchmark construction pipeline: real worklogs are segmented into tasks, anonymized with structure-preserving replacements, reconstructed into instructions plus containerized observations or controlled mock APIs, and aligned via rewritten trajectories so that the pre-task environment matches the original workflow state. This controlled trajectory alignment is what makes the memory gain attributable to recalled experience rather than to drift in earlier executions.","core_discovery":"The paper's discovery is that memory in language agents is best evaluated as downstream performance on real, longitudinally ordered workflows, not as retrieval accuracy. Under a fixed agent harness and model, adding recalled prior experience increases Workspace Score from 68.08 to 78.20 and Preference Score from 41.50 to 70.60 with the strongest memory component (a-mem), and the with-recall trajectory wins against the no-recall trajectory 72.70% of the time. Across five base models using mem0, recall improves both scores for every model, with Workspace gains ranging from +2.19 to +5.61 and Preference gains from +5.55 to +9.61. The mechanism it identifies is actionability: memory that preserves concrete paths, constraints, artifacts, and procedures lets agents reallocate effort from exploration to execution, while shorter summaries preserve conclusions but lose the working details needed to continue.","pith_inferences":["A natural extension is a fixed-token-budget comparison of summary versus in-context memory, which the paper itself flags as future work; if compact summaries perform on par when budgets match, the actionability advantage would shrink.","The diagnostic metrics (solvability, continuity, memory-induced rate) could be combined into a single memory-utility index that predicts downstream score gains, letting developers tune memory without running full Docker evaluations.","The benchmark's GPT-5.5 grading introduces a potential confound: observed memory gains may partly reflect that the grader prefers outputs resembling the reference trajectory, so human-calibrated rubrics would strengthen the causal claim.","The trajectory-alignment procedure may hide exactly the kind of environment drift that long-term memory is meant to cope with, so real-world gains could differ from benchmark gains."],"forward_implications":["Memory systems should be scored by whether they improve final workspace quality and preference alignment, not by retrieval metrics alone.","Curated in-context experience, meaning full task traces rather than distilled summaries, should be the stronger default for workflow-continuation tasks.","Models differ in how reliably they use recalled information, so a memory system that works for one base model may not transfer to another.","Because richer recall raises the rate of memory-induced problems, memory design needs to balance actionability against robustness to misleading recall.","The 568-task, 14-participant benchmark provides a reproducible testbed for comparing agent memory components and base models."],"supporting_citations":[{"why":"It is the closest prior benchmark for multi-session agent memory; ContextWeave directly contrasts its manually constructed dependencies with real-workflow dependencies.","marker":"[22]"},{"why":"Mem0 is the fixed memory component in the cross-model experiment, so the results across five base models depend on this implementation.","marker":"[27]"},{"why":"A-MEM is the in-context experience component that produces the highest Workspace (78.20) and Preference (70.60) scores in the fixed-model comparison.","marker":"[29]"},{"why":"LangMem is the structured-summary memory component whose intermediate results support the paper's claim that summarization underperforms in-context experience.","marker":"[28]"},{"why":"Codex is the agent harness used for all task executions and for workspace grading, so every reported score depends on this toolchain.","marker":"[31]"},{"why":"RULER represents the long-context retrieval style of memory evaluation that the paper argues is insufficient for measuring downstream agent performance.","marker":"[8]"},{"why":"LongMemEval is a main long-term conversational memory baseline that ContextWeave distinguishes itself from by measuring downstream task outcomes.","marker":"[11]"}],"fun_headline_variants":["Memory beats retrieval in real-world agent workflows","Agent memory lifts workflow scores by double digits","Recall powers agents through real office tasks","Actionable memory boosts agent performance in workflows","Experience-backed memory wins in longitudinal tasks"],"cache_read_input_tokens":42752,"weakest_assumption_plain":"The paper assumes that GPT-5.5 rubric grading is a trustworthy measure of workspace quality and participant-preference alignment; if that grading does not match human judgment, the reported memory gains may not reflect real usefulness.","fun_headline_variants_meta":{"raw":{"variants":["Memory beats retrieval in real-world agent workflows","Agent memory lifts workflow scores by double digits","Recall powers agents through real office tasks","Actionable memory boosts agent performance in workflows","Experience-backed memory wins in longitudinal tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000145,"raw_usage":{"total_tokens":1173,"prompt_tokens":931,"completion_tokens":242,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":178}},"tokens_in":547,"tokens_out":242,"duration_ms":3181,"temperature":1.0,"reasoning_tokens":178,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:31:58.622710+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 100 of the 568 core tasks, run the no-recall and strongest with-recall conditions, and have human annotators who do not know which output has memory blindly rate workspace usefulness and preference alignment; if human-rated gains are substantially smaller than the GPT-5.5-rated gains (68.08 to 78.20 and 41.50 to 70.60), the central claim that memory improves downstream performance in real workflows would be called into question.","supporting_citations":[],"review_version":1}