{"id":"e9c63465-3c24-4c63-95ba-cc68577e818a","arxiv_id":"2609.34603","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Repairing a failed episode improves its transfer value mainly by recovering successes the failed memory displaced, with benefits showing on one benchmark but not on two APEX regimes.","lead":"Does repairing a failed agent episode make it a better memory for the next task? This paper compares the same failure before and after accepted repair across 3,300 runs, finding gains on one benchmark, near-null results on two others, and that much of the apparent gain merely restores success the memory displaced.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"All headline quantities (+44/+29/+32pp gains; 22-of-46 restoration) are one-execution-per-cell point estimates, and the restoration/new-success label rests on one noisy B-only run per target; repeated ThinkingBox core-cell runs are needed before the empirical magnitudes can be trusted.","rationale":"The reader's weakest-assumption pick (one execution per cell, Section B.1 and b217) is exactly the load-bearing concern: I agree with it, so the verdict is unchanged. I verified that the decomposition arithmetic is correct: 44−29 = (67−64)−(23−35) = 3+12, and the transition counts in Table 11 sum to the net gaps (46 gains − 2 losses = +44 for ThinkingBox Full). The statistical reporting is internally consistent and carefully hedged: Table 7 shows all six ThinkingBox primary contrasts surviving global-24 at target level, while Table 8's group-level sensitivity does not, and the paper flags this explicitly. I also credit the transparency: full transition tables, leave-one-cluster-out ranges, frozen outcome records, recomputed interval sensitivity, and a candid disagreement audit. The central methodological conclusion — memory revisions must be evaluated against both predecessor and fresh start — is a design principle that holds regardless of any single cell's outcome, so my concern does not overturn the paper's conceptual contribution. What is conditional is the empirical demonstration: the ThinkingBox magnitudes and, especially, the 22-of-46 restoration interpretation, which assigns each target a restoration/new-success label based on one B-only run whose per-target probability is unestimated. Judging non-blindness (acknowledged at b581) and repairability-conditioned cohort selection (disclosed in Appendix A) are secondary because they mainly affect the already-null APEX contrasts and the scope, not the executable-assertion ThinkingBox results that carry the positive claims. Hence CONDITIONAL stands: accept the design principle, treat the empirical magnitudes as conditional on a repeated-run check of the ThinkingBox core cells.","tokens_in":29203,"tokens_out":13097,"duration_ms":164191,"concrete_test":"Run three independent repetitions of ThinkingBox B-only, U-Full, and C-Full on the same 100 pairs with the frozen Appendix A settings (900 additional B-stage executions). From the 3×3 cross-repetition table, compute the spread of the C–U gap (observed +44pp), the C–B-only gap (+25pp), and the restoration count n_101 (observed 22 of Full's 46 upward transitions); also estimate per-target w_i = 2p_i(1−p_i) and evaluate how much of the observed 46/2 discordance Eq. B.1 attributes to execution noise at equal probabilities. If the gaps spread by more than roughly ±10pp across repetitions, or n_101 ranges outside about 13–31, the empirical magnitudes and the restoration/new-success split are single-draw artifacts; the design principle survives, but the empirical section should be downgraded to indicative pending repetition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim has two parts: the design principle that C–U and C–B-only answer different questions and that both references are needed, and the empirical demonstration that ThinkingBox corrected memories beat both references (44/29/32pp correction, 25/22/18pp base) while APEX does not, with 22 of Full's 46 upward transitions restoring baseline success. The principle is logically secure; every empirical magnitude is a point estimate from one trajectory per cell (Section B.1; acknowledged at b217). The most exposed quantity is the restoration decomposition, which carries the abstract's 'distinct from the value of reusing it': each target is classified n_101 (restoration) or n_001 (new success) from a single B-only run, so for targets whose per-target success probability is near 0.5 the label is close to a coin flip. The 22-of-46 split could swing widely on the next set of runs, and with it the paper's headline interpretation. The same instability propagates into the aggregate gaps and into the '12 of 15' decomposition: the 12pp leg is U-Full (23) versus U-Skill (35), a contrast that does not survive family adjustment (Table 9: target pH = 0.3782). Eq. B.1 makes the mechanics explicit: Pr[Y(c)≠Y(c')] = (p_c−p_c')² + ½(w_c+w_c'), so a single run cannot separate task effects from execution variation. The paper's own Table 16 shows B-only observed outcomes differing across executions (37 pass in both regimes, 17 in exactly one), confirming single-run acceptance instability. Repetition is the one check that would tell whether the +44pp gap and the 22/46 split are persistent effects or single draws.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper studies whether updating an agent memory with an accepted repair of a failed source episode improves transfer to a fixed target task. Using 100 source–target pairs per regime across ThinkingBox, APEX-state, and APEX-text (3,300 runs, eleven conditions), each source is transferred both before (U) and after (C) accepted repair, alongside independent execution (B-only). The central design claim is that a memory revision must be evaluated against two references — its uncorrected predecessor (C–U, correction value) and a fresh start (C–B-only, baseline-relative value) — because a large correction gain can reflect a weak predecessor rather than a superior corrected memory, and an improvement can merely restore a success the memory itself displaced. Empirically, ThinkingBox shows large C–U and C–B-only gains (Full/Skill/Hybrid 44/29/32pp and 25/22/18pp), with 22 of Full's 46 upward transitions classified as baseline restoration; APEX shows no comparable aggregate benefit. The paper also separates a design principle (the two-reference requirement, which is logically sound) from the fragile empirical magnitudes, which all rest on one execution per cell.","tokens_in":29499,"tokens_out":4222,"duration_ms":26845,"significance":"If the empirical magnitudes were reliable, the paper would make a useful methodological contribution: the recommendation to evaluate any agent memory revision against both the previous-version memory and independent execution is clear, actionable, and backed by a sound paired same-episode design that holds provenance fixed. The paper is unusually transparent — it ships frozen records, reproducible scripts, and explicit statements of its own limitations (e.g., b217, b400, b433–b435, b581), and its decomposition claims are exact arithmetic on observed counts rather than fitted quantities. However, every headline magnitude is a point estimate from one execution per cell, a fact the paper acknowledges but does not fully propagate into the abstract and conclusions, which present the single-run values as settled effects. The contribution is therefore a defensible evaluation principle with an empirical demonstration of limited inferential strength.","major_comments":[{"comment":"The single-execution-per-cell design makes every headline magnitude — the +44/+29/+32pp correction gains, the 25/22/18pp baseline-relative gains, the 22-of-46 restoration split, and the 12-of-15 decomposition — a point estimate from one trajectory per condition, and Eq. (B.1) shows explicitly that a single run cannot separate a condition's task-specific effect from execution variation. The paper acknowledges this at b217 ('Group-level inference addresses task dependence, not repeat-run reliability'), and Table 16 confirms the instability: B-only outcomes differ across regime executions for 17 of 100 targets. Given that the abstract and Section 5 state these magnitudes flatly, the load-bearing empirical claims need either repeated runs on the core ThinkingBox and APEX cells or the magnitudes must be visibly re-framed throughout (including the abstract) as single-run observations whose sampling distribution is unknown.","section":"B.1, Eq. (B.1); Sections 5.1–5.3 and abstract","authors_acknowledge":"Group-level inference addresses task dependence, not repeat-run reliability; repeti- tions and blinded artifact adjudication would distinguish persistent gains from execution and scoring variation.","authors_acknowledge_p":0.89},{"comment":"The claim that 22 of Full's 46 upward transitions 'restore observed baseline success' is a central empirical support for the paper's thesis that repairing experience is distinct from reusing it, but each target is classified n_101 (restoration) versus n_001 (new success) from a single B-only run. For targets whose per-target acceptance probability is near 0.5, Eq. (B.1) shows this label is close to a coin flip, so the 22-of-46 split could swing widely on repeated runs. The paper's own Table 16 shows exactly this fragility for B-only outcomes. The restoration/new-success distinction should be presented as a within-this-run observation, not as a measured property of the condition, or supported by repeated B-only runs on the relevant targets.","section":"Section 5.2, Fig. 3, Table 11; abstract","authors_acknowledge":"Group-level inference addresses task dependence, not repeat-run reliability; repeti- tions and blinded artifact adjudication would distinguish persistent gains from execution and scoring variation.","authors_acknowledge_p":0.76},{"comment":"The conclusion that Full's 15-point larger correction gap 'mostly reflects its weaker starting point' (12 of the 15 points from U-Full's worse performance) rests on the U-Full versus U-Skill contrast of 35 vs 23 targets, which does not survive even target-level family adjustment (Table 9: U-S–U-F target pH = 0.3782). The paper hedges only the 3-point C-Full advantage ('not established by the adjusted representation tests') but not the 12-point U leg. Since both legs are single-run point estimates and the U contrast is not statistically established, the interpretive claim that Full's apparent superiority is an artifact of a weaker predecessor should be stated as an arithmetic observation on this run's data, not as a robust finding.","section":"Section 5.3, Table 9","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The abstract says inference 'weakens at the task-family level,' while Section B.2 states that no primary contrast survives the group-level global-24 adjustment. The abstract should state the stronger fact directly: all headline correction effects fail to survive the combined group-level and family adjustment, even though the ThinkingBox target-level effects survive the global-24 target-level adjustment.","section":"Abstract and Section B.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The distinction between this study and prior negative-transfer work (Feng et al. 2026; Li et al. 2026a) would be clearer if the paper explicitly noted that the two-reference evaluation (C–U and C–B-only) is its novel axis, and briefly stated whether any prior benchmark reported the restoration-vs-new-success decomposition on a same-episode pair.","section":"Section 2 and Appendix G","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The sensitivity layer that changes Text Execution–Summary from 52/40 with 14/2 discordances to 50/38 with 12/0 discordances is presented clearly, but the retention of 'first valid judgments' as the main result should be flagged in the main text's Section 6.1 discussion, where the +12pp figure is stated without reference to this audit.","section":"Appendix F.3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Several tables mix percentage-point gaps with raw counts in the same display (e.g., Fig. 2, Tables 7–8). A consistent notation — e.g., always placing 'pp' adjacent to gaps and leaving counts unlabeled — would reduce the risk of misreading the exploratory intervals as error bars on the counts.","section":"Throughout","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The new thing here is the axis, not the method. Comparing the same failed episode before and after accepted repair, on a fixed target, against both the uncorrected predecessor (C-U) and independent execution (C-B-only), is a real addition to the ExpeL/CLIN/AWM lineage. The restoration-versus-new-success decomposition (101 vs 001) is the useful part: it makes explicit that a correction gain can just reflect a weak predecessor, and that half of Full's upward transitions only recover a success the memory itself displaced. That distinction is worth keeping regardless of the magnitudes. The '12 of 15 points from worse U' point is also a good cautionary result. Credit where due: the paper is unusually honest. Section B.1 states plainly that one execution per cell cannot separate task effects from execution variation, b217 flags the repeat-run reliability issue, and the non-blind audits and cohort conditioning on repairability are all disclosed. The arithmetic checks out - I verified the 22-of-46 split and the 12-of-15 decomposition against Table 11. No fitted parameters, no circularity. The soft spots are real but proportionate. First, every headline quantity - +44/+29/+32pp, the 22/46 restoration split - is a point estimate from one trajectory per cell. For targets near p = 0.5, the B-only classification that determines whether a gain is 'restoration' or 'new success' is close to a coin flip, and Table 16 shows B-only outcomes shifting across executions. The 22-of-46 split could swing noticeably on the next run set. Second, none of the 24 primary contrasts survive the global-24 adjustment; the ThinkingBox effects hold at target-level family adjustment but not group-level. The paper reports this honestly, but it means the empirical case is conditional, not established. These are acknowledged limitations, and I am not pretending to have found them - but they are load-bearing for the magnitude claims. What survives all this is the principle: evaluate a memory revision against both its predecessor and a fresh start. That is a solid, citable design contribution that does not depend on the point estimates. The empirical magnitudes need repeated core-cell runs and blinded audits before they should be quoted as effects. This deserves a serious referee. It is new, transparent, and the methodological conclusion stands even if the empirical chapters need revision. I'd engage with it, and I'd cite the design principle in my own work. The reviewer's job is to insist on the repetition, not to reject the question.","headline":"Genuinely new pre/post-repair same-episode axis and a durable design principle, but headline magnitudes rest on one execution per cell and fail global adjustment.","tokens_in":30677,"tokens_out":932,"duration_ms":21267},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Repairing an episode and reusing it are distinct: every memory update should be judged against both the failed version it replaces and a fresh start, since a big correction gain can just reflect a weak old memory or a restored success.","keywords":["agent memory","experience reuse","repaired episodes","memory update evaluation","transfer of corrected experience","workflow benchmarks","LLM agents","acceptance decomposition"],"falsifier":"Repeat the experiment on the same 100 ThinkingBox pairs, executing U-Full, C-Full, U-Skill, C-Skill, and B-only several times per target and comparing acceptance rates and outcome patterns across repetitions. If the 23% versus 35% uncorrected gap (the source of twelve of Full's fifteen extra points) or the 22-of-46 baseline-restoration count moves by more than a few percentage points, the paper's decompositions would have to be reported as ranges rather than point estimates; if the numbers are stable, the empirical support for the two-reference principle holds.","tokens_in":30700,"feed_emoji":"🔧","tokens_out":20419,"duration_ms":222270,"temperature":0.7,"pith_summary":"Most agent-memory systems assume that once a failed run is repaired and accepted, the repaired episode is the better experience to carry into the next task. This paper argues that repairing an episode and reusing it are different achievements, and that the repaired memory must be tested against two references at once: the uncorrected failure it replaces and an independent run of the next task with no source memory at all. In the ThinkingBox workflow benchmark the corrected full-history handoff beats the failed original by 44 percentage points, but twelve of the fifteen points of its advantage over the skill abstraction come from the failed full-history handoff being worse, not from the fix being better, and 22 of its 46 improvements only restore successes that an independent run already achieved. In the two APEX regimes the same comparisons show no reliable aggregate correction benefit, and text-Hybrid is directionally negative. The paper's conclusion is that a memory update should be judged by both a correction gain and a fresh-start gain, because a large correction gain alone can reflect a weak old memory rather than a superior repaired one.","feed_headline":"Bigger repair gains can hide weaker old memories","feed_subtitle":"Twelve of a fifteen-point advantage came from a worse failed memory, not a better fix.","key_machinery":"The load-bearing object is a paired three-branch design: the same failed episode is transferred to the same fixed target as an uncorrected record (U), as the first accepted repaired record (C), and not at all (B-only), with the same 100 source–target pairs serving every condition in each regime. The argument is carried by an outcome-pattern decomposition, $n_{buc}$, the number of targets whose B-only/U/C acceptance bits are $(b,u,c)$, which makes the net correction gain $N\\,\\widehat{\\Delta}^{corr} = n_{001} + n_{101} - n_{010} - n_{110}$. That identity is what separates 'new success beyond both references' ($n_{001}$) from 'restoration of a success the old memory displaced' ($n_{101}$), and it is what lets the paper show that a large correction gain can be mostly a weak uncorrected starting point rather than a strong fix, converting an apparent 44-point win into a decision-relevant breakdown.","core_discovery":"The paper's central claim is that the value of repairing an episode and the value of reusing it are separate quantities, so an accepted repair should not automatically replace the failed memory it came from. The authors test this with a same-source, fixed-target design: for each of 100 source–target pairs in each of three regimes, the same failed source A is handed to the same new task B in an uncorrected state (U), after its first accepted repair (C), and not at all (B-only), holding provenance and target fixed. The evidence is that correction gains are easy to overstate in one regime and absent in another: in ThinkingBox, C beats U by 44/29/32 points for Full/Skill/Hybrid, yet twelve of Full's fifteen extra points over Skill come from U-Full's weaker starting point, and 22 of Full's 46 upward transitions only restore a B-only success that the uncorrected memory had displaced; in both APEX regimes no correction gain is established, with text-Hybrid at −8 points. The paper concludes that reporting only a correction gain conflates a weak predecessor with a strong fix, that neither the predecessor comparison nor the fresh-start comparison alone is enough, and that memory design should preserve the conditions under which a repair's advice was valid.","pith_inferences":["Editorial extension: the two-reference principle transfers to any system that swaps an old artifact for a repaired one — a fine-tuned model, a rewritten prompt, an updated skill library — where an apparent improvement over the old artifact should likewise be checked against a fresh baseline so that restorations are not counted as new gains.","Editorial extension: the outcome-pattern decomposition $n_{001}, n_{101}, n_{010}, n_{110}$ would be a cheap standard diagnostic for agent-memory benchmarks, letting readers separate newly unlocked successes from restored successes without re-running experiments.","Editorial extension: the paper's memory-design sketch — retain what changed, the requirement that justified the change, and the inputs it depended on — implies a concrete testable artifact: store each repair with an applicability predicate evaluated against the new request, and check whether the APEX convention-conflict cases (such as mid-year versus full-year discounting) stop failing."],"forward_implications":["Any evaluation of an agent-memory revision should report both a correction gain (corrected memory versus the uncorrected version it replaces) and a fresh-start gain (versus independent execution), because a large correction gain alone can reflect a weak predecessor rather than a superior corrected memory, as when twelve of Full's fifteen extra points over Skill came from U-Full being worse, not C-","A corrected memory that beats its predecessor can still lose to a fresh start: 22 of Full's 46 upward transitions only restored a B-only success that the old memory had displaced, so adopting a repair should hinge on beating independence, not merely on beating the failure.","Target-level significance does not establish family-level generality: under group-level sign-flip sensitivity, no primary contrast survives global adjustment, so the ThinkingBox gains weaken when task families rather than individual targets are the unit of analysis.","The benefit does not generalize across regimes: neither APEX set establishes aggregate correction gains, and APEX-text's accepted-Execution advantage (52% versus 40% for Summary) is only five points above B-only's 47%.","Smaller handoffs trade one cost for another: Execution cut cumulative input by 12–32% while increasing model calls in every regime, so memory selection must weigh success, context reading, and downstream interaction demand jointly."],"supporting_citations":[{"why":"Supplies ThinkingBox, the stateful workflow benchmark whose executable validators generate all of the paper's ThinkingBox acceptance and lifecycle data.","marker":"Li et al., 2026b"},{"why":"Supplies APEX-Agents, the benchmark behind both the APEX-state and APEX-text regimes and their shared 100 source–target pairs.","marker":"Vidgen et al., 2026"},{"why":"Self-Refine, the iterative-feedback refinement paradigm that the accepted-repair branch (C) instantiates.","marker":"Madaan et al., 2023"},{"why":"Reflexion, the prior approach converting task feedback into improved later attempts, which the paper's post-acceptance transfer setup extends.","marker":"Shinn et al., 2023"},{"why":"ExpeL, the experience-extraction and reuse method the paper contrasts with its same-episode pre/post-repair comparison.","marker":"Zhao et al., 2024"},{"why":"Agent Workflow Memory, the workflow-extraction approach whose stored procedures resemble the paper's Skill handoff.","marker":"Wang et al., 2025"},{"why":"ReasoningBank, which learns from both successes and failures and motivates the question of whether a repaired episode is a better memory.","marker":"Ouyang et al., 2026"}],"fun_headline_variants":["Repair gains may just mean the old memory was worse","Fixing a memory isn't the same as reusing it","Correction wins can hide a weak failed baseline","A bigger fix score can come from a worse past","Repair value decoupled from reuse value in agent memory"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The load-bearing premise is that one execution of each condition on each target gives a trustworthy estimate of that condition's acceptance behavior; the paper acknowledges that its resampling accounts for task-composition uncertainty but not repeated-run reliability, so the headline numbers — the 44-point ThinkingBox gain, the 23-versus-35 starting gap behind Full's fifteen extra points, and the 22-of-46 restoration count — could shift if the same cells were run again.","fun_headline_variants_meta":{"raw":{"variants":["Repair gains may just mean the old memory was worse","Fixing a memory isn't the same as reusing it","Correction wins can hide a weak failed baseline","A bigger fix score can come from a worse past","Repair value decoupled from reuse value in agent memory"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000154,"raw_usage":{"cost":0.000154168,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000154168,"upstream_inference_prompt_cost":0.000114968,"upstream_inference_completions_cost":3.92e-05},"total_tokens":1166,"prompt_tokens":1026,"completion_tokens":140,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":58}},"tokens_in":1026,"tokens_out":140,"cost_source":"openrouter_usage","duration_ms":2351,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":58,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-30T01:31:40.319419+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Repeat the experiment on the same 100 ThinkingBox pairs, executing U-Full, C-Full, U-Skill, C-Skill, and B-only several times per target and comparing acceptance rates and outcome patterns across repetitions. If the 23% versus 35% uncorrected gap (the source of twelve of Full's fifteen extra points) or the 22-of-46 baseline-restoration count moves by more than a few percentage points, the paper's decompositions would have to be reported as ranges rather than point estimates; if the numbers are stable, the empirical support for the two-reference principle holds.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":2}