{"id":"4790dd3b-b81e-404f-b151-a8d1c703bd62","arxiv_id":"2608.22533","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Contrasting multiple models' attempts at the same task yields a compact, transferable procedural memory that more than doubles held-out success for computer-use agents.","lead":"The paper presents CONTRAMEM, a training-free system that learns reusable procedural memory by comparing successful and failed attempts at the same task from different AI models. On computer-use benchmarks, the memory more than doubles held-out task success and transfers to unseen models, suggesting that cross-model disagreement is a useful supervision signal.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed mechanism rests on unmeasured Reflector/Curator attribution accuracy; end-to-end gains do not rule out useful-but-spurious generic rules as the active ingredient.","rationale":"I reviewed the strongest claim against the evidence. The empirical results are internally consistent: paired held-out evaluations, a matched multi-rollout control (Table 3), a weaker construction model on AppWorld (GPT-4.1-mini), explicit privacy and leakage validations, and honest reporting of the Time limitation all support the headline success rates. The weakest link is not the evaluation itself but the interpretive mechanism: the paper asserts that same-task contrast exposes the causal decision boundary, yet the only enforcement is prompt-level constraint in the Reflector/Curator, with no direct measurement of attribution accuracy. Because the bank is built once and reused, a systematic misattribution would be inherited everywhere. The self-memory and multi-rollout controls vary trajectory inputs, not attribution correctness, so they cannot rule out an alternative explanation: the distiller may be producing useful generic procedural guidance that helps regardless of whether the specific attributed boundary is causal. End-to-end transfer provides indirect evidence, but a strong LLM distiller with generic prior knowledge could yield helpful text even from spurious attributions, so the end-to-end result conflates attribution quality with generic helpfulness. This does not overturn the empirical claim; it narrows what is proven. The reader's weakest assumption identifies the same issue, and the proposed synthetic probe would settle it by measuring attribution precision and recall against known decision boundaries. The verdict remains conditional: accept the empirical results as credible, but require the attribution probe (and ideally code/data release) before the mechanistic claim is fully verified.","tokens_in":28808,"tokens_out":9537,"duration_ms":92560,"concrete_test":"Run a synthetic attribution probe on 50 GAIA2-style tasks where, for each task, the success/failure trajectories are scripted to differ in exactly one known procedural decision (e.g., completing the safe branch vs asking immediately; enumerating the full target set vs skipping one write), with verifier outcomes assigned deterministically from that decision. Feed these packets through the exact Reflector and Curator prompts in Appendices B.4–B.5 and measure how often the emitted Skill Card's core_rule and completion_condition match the manipulated boundary (precision/recall against known ground truth) and how often it introduces rules not supported by the packet. High recall with few spurious rules would resolve the concern; low recall would mean the end-to-end gains need a different mechanistic explanation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that same-task outcome variation is procedural supervision and that the Reflector/Curator (Appendix B.4–B.5) distills the causal decision boundary between success and failure into Skill Cards. The load-bearing, unverified step is attribution reliability: the Reflector is prompted to 'locate the decision boundary' and 'attribute the outcome,' and the Curator is constrained by semantic closure, but the correctness of the extracted boundary is never measured against an oracle. Because the bank is built once and reused for every target, any systematic misattribution is inherited by all downstream evaluations. The existing controls (self memory, matched multi-rollout in Table 3, GPT-4.1-mini construction on AppWorld) vary trajectory inputs or construction model, but they do not test whether emitted cards encode the actual causal contrast rather than useful generic procedure. An alternative account is consistent with the data: the LLM distiller may produce helpful transferable advice ('verify before destructive writes', 'ask before ambiguous writes') that improves agents even when the specific attributed boundary is spurious. Table 1's Time results, where Claude Sonnet's self memory is 10.0% but ContraMem is 2.5% and contrastive memory does not beat self memory overall on Time, show the method's limits but do not address attribution quality. Thus the mechanistic half of the strongest claim is supported only end-to-end, not directly; if attribution accuracy is low, the explanation of why ContraMem works would need substantial revision even though the success rates might stand.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ContraMem, a training-free, source-flexible framework that treats same-task outcome variation across multiple model trajectories as procedural supervision. For each reference task, a Reflector compares the contrasting trajectories and emits candidate Skill Deltas; a Curator applies localized edits (ADD, PATCH, MERGE, NARROW, NOOP) to a bank of app-level Function Cards and task-level Skill Cards; at test time a deterministic retriever injects a small task-relevant subset into a single target agent. The authors report that on held-out GAIA2/ARE tasks, ContraMem raises the source-target macro success rate from 26.2% to 55.3%, transfers the same bank to an unseen Qwen3.7 Plus (18.5% to 35.5%), and outperforms both no-memory and self-memory controls on AppWorld for three mid-tier agents on both public test splits. Ablations compare self memory, same-model multi-rollout, single-card-type variants, and adapted prior memory systems, with McNemar significance tests, efficiency measurements, construction-cost accounting, and an honest discussion of the Time ability's residual difficulty.","tokens_in":29121,"tokens_out":6888,"duration_ms":65698,"significance":"If the empirical results hold, this is a substantial contribution to training-free procedural memory for computer-use agents. The idea of using cross-model disagreement as a supervision signal is clean, and the evaluation is unusually careful: matched-budget multi-rollout controls, self-memory controls, unseen-model transfer, frozen-bank runtime, cross-benchmark validation, and transparent construction-cost accounting are all present. The AppWorld construction using GPT-4.1-mini as the Reflector/Curator is a good control against the concern that gains come from a stronger construction model. The paper also reports its main limitation honestly: text-injected memory does not resolve time-sensitive failures, and Time results show only small or mixed gains. The principal weakness is that the mechanistic claim about distilling the causal decision boundary is supported only end-to-end; the experiments do not directly measure whether emitted cards encode the true causal contrast rather than useful generic heuristics. This is a load-bearing gap for the paper's framing, though not for the more modest claim that the memory bank improves held-out performance.","major_comments":[{"comment":"The load-bearing assumption is that the Reflector can correctly locate the causal decision boundary separating success from failure in a contrast packet and that the Curator preserves this boundary under semantic-closure edits. This assumption is never directly evaluated. The existing controls vary trajectory inputs (self memory vs. multi-model), the construction model (GPT-4.1-mini on AppWorld), and the memory type (Table 3), but none measures whether a card's core_rule corresponds to the actual causal contrast in the packet. The alternative account that the pipeline distills useful but generic advice ('verify before destructive writes', 'ask before ambiguous writes') is consistent with the data, because such advice would also improve held-out success. I recommend one of: (a) a sampled oracle evaluation of Reflector attribution against an independently annotated decision boundary; (b) a card-level flip analysis showing which cards drive the failure-to-pass and pass-to-fail transitions; or (c) rewording the mechanistic claims from 'the causal decision boundary' to 'transferable procedural guidance' wherever they appear in the abstract and methodology. As written, the causal-attribution half of the central claim is supported only end-to-end.","section":"Methodology, 'Contrastive Skill Memory' (Reflector and Curator paragraphs; Appendix B.4–B.5)"},{"comment":"The retrieval configuration contains several free parameters — the BM25 soft app-family prior weight lambda_1, the retrieval caps (top-3 Skill Cards, top-2 Function Cards), and the AppWorld task-start memory budget — but the paper provides no sensitivity analysis and does not state how these values were selected. If these parameters were chosen after inspecting held-out performance, the reported gains could be inflated; if they were fixed a priori, that should be stated explicitly. Please report the selection procedure and provide a small sensitivity sweep (e.g., lambda_1 in {0, 0.1, 0.5}, Skill/Function caps in {1, 2, 3, 5}) on at least one ability or benchmark to show that the main conclusion is robust to reasonable variation in these settings.","section":"Appendix A.8, Eq. (1); Table A1"},{"comment":"The headline comparison to AWM and ACE is against evidence-matched adaptations of those systems, not against the original published implementations or their original benchmark scores. The appendices document the adaptations carefully and honestly, but the main text's claim of being '30.0 points above the strongest prior memory system' (Introduction, repeated in the Abstract and Conclusion) and the Figure 3 label do not carry this caveat. This is important because a reader could reasonably interpret the number as a direct comparison to the published systems' results. Please qualify the claim in the main text, for example by writing 'strongest prior memory system under our evidence-matched adaptation' or by moving the exact comparison to the appendices and using a more hedged phrase in the abstract.","section":"Figure 3; Appendices A.6–A.7"}],"minor_comments":[{"comment":"ContraMem's Time success (2.5%) is below the self-memory control (10.0%) for Claude Sonnet, and the three source targets' Time aggregate is essentially flat. The experimental discussion is honest about this, but the abstract's unqualified 'beats ... self memory' formulation and Figure 1's 'beats self memory on every metric' should be qualified to macro-level metrics or explicitly note per-cell Time regressions.","section":"Table 1, Claude Sonnet Time row"},{"comment":"The phrase 'the same construction carries over unchanged to AppWorld' overstates the actual transfer: the AppWorld instantiation uses a deterministic task signature instead of an ability label, a doc-delta rule for API cards, a different Reflector focus block, a different construction model (GPT-4.1-mini), and a different injection budget (Appendices B.7–B.11). The algorithmic recipe is the same, but the instantiation is benchmark-aware; please rephrase to 'the same algorithmic recipe' or state the adapters explicitly where the claim first appears.","section":"Abstract and Experiments, AppWorld validation"},{"comment":"No code release is mentioned. Releasing the bank construction pipeline, retrieval scoring, and frozen-bank configuration would substantially improve reproducibility, especially given the number of prompt templates and benchmark-specific adapters in Appendices A and B.","section":"General reproducibility"},{"comment":"The table formatting appears corrupted in several cells: for Claude Sonnet, '52.552.5' should presumably be '52.5' (self) and '57.5' (ContraMem), and '10.051.0' should be '10.0' and '51.0'; for DeepSeek V4 Pro, '77.532.5' should be '77.5' and '32.5'. Please regenerate the table to ensure each cell contains a single value.","section":"Table 1"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know one thing up front: this is a serious empirical paper, not a flashy sketch. The core idea—treating same-task outcome variation across models as supervision, then distilling it into typed Function/Skill Cards through a local curator—is genuinely new as a combination, and the evaluation is considerably more careful than the field's average. The headline results (26.2% to 55.3% held-out on GAIA2/ARE, transfer to unseen Qwen3.7 Plus, consistent gains on AppWorld) come with matched multi-rollout controls, a self-memory control inside the same architecture, McNemar tests, and honest reporting of the Time limitation. The appendix is a real attempt at reproducibility: full prompts, schemas, baseline adaptation details with fidelity caveats, runtime accounting. The AppWorld construction using GPT-4.1-mini as Reflector/Curator is a nice control against the objection that a stronger model is doing the work.\n\nThe soft spots are real but mostly in the interpretation, not the headline. The stress-test concern lands: the Reflector/Curator attribution accuracy is never measured directly. The paper does not show that the emitted cards encode the actual causal decision boundary rather than useful generic advice ('verify before destructive writes', 'ask before ambiguous writes'). The existing controls—self-memory still helps, multi-rollout trails contrastive—do rule out source-strength and sample-count explanations, but they do not rule out the possibility that the bank is a better collection of generic heuristics. If attribution is often wrong, the explanation of why ContraMem works would need revision, even though the success rates might hold. To me this is a moderate concern about the mechanism paper, not a fatal flaw in the method.\n\nTwo smaller items: no code or data release, which matters for a system like this; and no direct comparison with MemCollab, the closest prior work, although the difference in goals is stated clearly. The retrieval hyperparameters (BM25 prior, caps) are fixed without sensitivity analysis—minor.\n\nWho is this for? Anyone working on agent memory, self-improving agents, or computer-use evaluation. It deserves a serious referee: a good reviewer will push for an attribution probe (e.g., hand-label a sample of cards as causal vs. generic) and for code release, but the paper's central empirical claim is supported. I would cite it and would bring it up in a reading group focused on agent memory.","headline":"A serious, well-validated paper on training-free procedural memory; the headline result survives scrutiny, but the causal-attribution mechanism is only supported end-to-end, and the paper deserves peer review with an eye on code release and a direct attribution probe.","tokens_in":29666,"tokens_out":1451,"duration_ms":15866},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Same-task outcome differences across models, distilled into typed memory cards, more than double held-out agent success.","keywords":["procedural memory","computer-use agents","contrastive trajectory distillation","multi-model trajectories","Function Cards","Skill Cards","GAIA2/ARE","AppWorld"],"falsifier":"Construct contrast packets with a known decision boundary by taking one successful trajectory and injecting a single flipped step into otherwise identical copies, then measure how often the Reflector names that step as the boundary; if attribution accuracy is near chance, the bank's later success must be re-attributed to generic instruction effects rather than the extracted card rules.","tokens_in":28621,"feed_emoji":"🧠","tokens_out":8497,"duration_ms":67828,"temperature":0.7,"pith_summary":"The paper tries to establish that an autonomous computer-use agent can build reusable procedural memory without retraining by treating the differences between several models attempting the same task as supervision. It claims that distilling the boundary between success and failure into compact cards—global Function Cards describing how tools behave and ability-level Skill Cards describing decision rules—and maintaining the bank through small local edits raises held-out success on GAIA2/ARE tasks from 26.2% to 55.3% on average across three source models, and that the same untouched bank lifts an unseen model from 18.5% to 35.5%. The construction also beats both no-memory and single-source self-memory on AppWorld for three mid-tier agents, which points to contrastive behavioral diversity rather than extra sampling as the source of the gain.","feed_headline":"Multi-model contrast doubles agent success to 55.3%","feed_subtitle":"Distilled from same-task outcome differences, a training-free card bank lifts an unseen model to 35.5%.","key_machinery":"The load-bearing object is the task contrast packet: one reference task's trajectories from multiple source models, normalized into a common event sequence and grouped so that outcome, action spans, writes, and verifier feedback can be compared side by side. Around it sit the two memory representations—Function Cards, which record observed callable-function contracts, side effects, and misuse guards, and Skill Cards, which record the distilled decision boundary with a trigger, core rule, and completion condition. The Reflector reads packets to locate the boundary and emits at most three candidate deltas; the Curator edits the existing bank through localized ADD, PATCH, MERGE, NARROW, or NOOP operations, and a deterministic retriever injects a small task-relevant subset at runtime.","core_discovery":"The central claim is that same-task outcome variation is a supervision signal: when several models attempt one reference task, their differences in correctness, efficiency, recovery, and failure modes expose the procedural decision boundary that explains why some runs succeed and others fail. ContraMem encodes those boundaries as typed memory rather than raw traces: Function Cards capture evidence-grounded tool contracts, Skill Cards capture transferable rules with explicit completion conditions, and a Curator applies the smallest valid edit (Add, Patch, Merge, Narrow, No-op) so the bank stays compact. The paper reports that this bank more than doubles source-target macro success on held-out GAIA2/ARE tasks (26.2% to 55.3%), transfers unchanged to an unseen target, and beats no-memory and self-memory baselines on both AppWorld public test splits; controlled ablations attribute the gain to heterogeneous contrast rather than sampling volume or source strength.","pith_inferences":["A testable consequence the paper leaves implicit: if attribution is sound, then contrast packets are a free self-supervision signal, and a direct probe would be to hold out outcome labels from the Reflector and measure how much of the gain remains.","The paper's own Time-ability result (self and contrastive memory within 10/120 vs 9/120) suggests a boundary: text-injected memory repairs failures it has encoded but not asynchronous state tracking, pointing to a hybrid of curated memory with a runtime obligation controller.","Transfer to an unseen model could be pushed further by evaluating open-ended deployment with the ability label removed, letting retrieval select across all banks and testing whether app-independent rules survive cross-family transfer.","One could falsify the causal claim by reversing an extracted decision rule in a card (e.g., ask-then-write instead of write-then-ask) and checking whether the target agent flips its behavior as predicted; if it does not, the card is not what drives the outcome."],"forward_implications":["Held-out success on GAIA2/ARE more than doubles from 26.2% to 55.3% averaged across the three source targets, with every target gaining (GPT-5.5 +33.5, Claude Sonnet 4.6 +24.5, DeepSeek V4 Pro +29.5).","The shared bank transfers unchanged to Qwen3.7 Plus, a target that contributed no trajectories, raising its success from 18.5% to 35.5%.","On AppWorld, the same construction beats both no memory and target-specific self-memory for all three mid-tier agents on both public test splits, with the largest margins in scenario completion (60.1% vs 44.6% for self-memory on Test-Normal).","Under a matched trajectory budget, three heterogeneous models' trajectories outperform three same-model rollouts (77.5% vs 70.0% macro), indicating contrastive behavioral diversity is the active ingredient.","Localized curation removes 24.9% of the skill cards while raising GPT-5.5 held-out success from 65.0% to 71.9%, so the gain comes from sharper cards, not more memory."],"supporting_citations":[{"why":"Defines the GAIA2/ARE benchmark and its verifier, which supplies the held-out tasks and success labels in the main evaluation.","marker":"Froger et al. 2026"},{"why":"Defines AppWorld, the cross-benchmark testbed with its public TGC/SGC splits used for validation.","marker":"Trivedi et al. 2024"},{"why":"Supplies the Agent Workflow Memory baseline that the paper reproduces offline and must outperform.","marker":"Wang et al. 2024"},{"why":"Supplies the ACE evolving-playbook baseline and the incremental generation-reflection-curation updating idea ContraMem contrasts with.","marker":"Zhang et al. 2025"},{"why":"Closest prior work that also contrasts same-task trajectories across models, but for agent-invariant transfer; ContraMem positions its disagreement-driven memory against it.","marker":"Chang et al. 2026"},{"why":"Represents the single-agent trajectory-memory line whose self-memory control isolates what heterogeneous contrast adds.","marker":"Fang et al. 2026"},{"why":"Provides the paired proportions test used to establish statistical significance of failure-to-pass flips.","marker":"McNemar 1947"}],"fun_headline_variants":["Contrasting multi-model outcomes double agent success without training","Training-free memory: contrast teaches agents, success doubles","Same-task contrasts build memory that lifts success to 55%","Outcome contrasts become memory, doubling computer-use success","Transferable memory from contrasts doubles agent success"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline rests on the assumption that the reflecting model can reliably identify which actual decision separated success from failure in a batch of same-task runs, and that the curator's edits do not introduce spurious rules; if attribution is wrong, the cards encode correlations rather than causes.","fun_headline_variants_meta":{"raw":{"variants":["Contrasting multi-model outcomes double agent success without training","Training-free memory: contrast teaches agents, success doubles","Same-task contrasts build memory that lifts success to 55%","Outcome contrasts become memory, doubling computer-use success","Transferable memory from contrasts doubles agent success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000736,"raw_usage":{"total_tokens":3349,"prompt_tokens":1062,"completion_tokens":2287,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":678,"completion_tokens_details":{"reasoning_tokens":2210}},"tokens_in":678,"tokens_out":2287,"duration_ms":16214,"temperature":1.0,"reasoning_tokens":2210,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-27T18:24:32.524820+00:00","model_set":{"reader":"deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.0-alltime-deepseek-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Construct contrast packets with a known decision boundary by taking one successful trajectory and injecting a single flipped step into otherwise identical copies, then measure how often the Reflector names that step as the boundary; if attribution accuracy is near chance, the bank's later success must be re-attributed to generic instruction effects rather than the extracted card rules.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}