{"id":"82d8d452-cca3-4443-9962-223f89fae256","arxiv_id":"2611.17068","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Evolutionary search over complete memory pipelines produced conversational-memory programs that beat manual baselines, with raw-history retention and query-aware evidence composition as transferable design elements.","lead":"The authors used an AI-driven search that rewrites whole memory programs from nearly empty starting points, and the evolved programs beat manually designed memory systems on two conversational benchmarks. The useful findings are two design habits: keep raw conversations available and tailor how retrieved evidence is packaged to the type of question.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Evaluation-set program selection may inflate reported gains; a held-out test set is needed to confirm that the selected programs generalize beyond the 80% evaluation partition.","rationale":"The reader's weakest_assumption focuses on the representativeness of the 20/80 split and potential bias in evaluation-set selection. My concern is more specific: the evaluation set is used for final program selection (Appendix B.2: 'For each setting, we report the remaining program with the highest evaluation-set accuracy'). This creates a selection-on-the-test-set issue that is not fully addressed by the paper. The cross-benchmark transfer experiments (Tables 6, 7) provide some evidence of generalization, but they do not test within-benchmark generalization to a truly held-out sample. The paper's limitations mention that 'evolutionary trajectories and resulting programs can vary across runs' but do not discuss evaluation-set overfitting. I agree with the reader that the split is a concern, but I identify a more precise mechanism: the evaluation set serves as both selection and reporting set, which can inflate reported accuracy. My concrete test proposes a simple fix: hold out a further test set to verify that the selected program's accuracy is not an artifact of selection. I recommend keeping the verdict as CONDITIONAL because the concern is real but does not invalidate the paper's contributions; it merely suggests that the reported numbers should be interpreted with caution until a proper held-out test is performed.","tokens_in":26278,"tokens_out":707,"duration_ms":101179,"concrete_test":"Re-run the evaluation on a fresh held-out test set (e.g., an additional 10% of data not used in search or evaluation selection) for LoCoMo and LongMemEval. If the accuracy of the selected programs drops by more than 2-3% absolute compared to the reported evaluation-set accuracy, then the selection procedure has overfit to the evaluation partition and the headline gains are inflated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim that evolved programs outperform human-designed baselines rests on accuracy numbers reported on the evaluation set (80% of data). However, the final program for each setting is selected as the one with highest evaluation-set accuracy among shortlisted candidates (Appendix B.2). This means the evaluation set doubles as a selection set for the final reported program. While the search set (20%) is held separate during evolution, the evaluation set is used for both selection and reporting. If the evaluation set is not representative or if selection overfits to its particular composition, the reported gains may overstate performance on a truly held-out sample. The paper acknowledges that 'program selection and design interpretation involve empirical judgment' (Limitations), but does not discuss the risk of evaluation-set overfitting. The reader's weakest_assumption correctly identifies the fixed 20/80 split as a concern, but the deeper issue is that the 80% evaluation set is not a pure test set—it is used for final selection. A proper held-out test set (e.g., a further 10% split) would be needed to verify that the selected program's accuracy is not an artifact of selection on that particular partition. The paper's cross-benchmark transfer experiments provide some evidence of generalization, but they do not address within-benchmark overfitting to the evaluation set's specific questions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper introduces MAPE, an evolutionary framework that treats an agent memory system as a whole program over a lifecycle interface (build, retrieve, compose, answer) and evolves it from minimal seeds via an LLM proposer and a diagnosis loop. The authors run evolution on LoCoMo and LongMemEval-S under accuracy-only (MAPE-A) and accuracy-plus-cost (MAPE-P) objectives, then select final programs by shortlisting on search accuracy or the accuracy-cost Pareto frontier and picking the best on a disjoint 80% evaluation partition. They report that all selected programs beat the evaluated human-designed baselines on their respective benchmarks, with up to 12% absolute accuracy gain at 35% of serving cost on LoCoMo under backbone transfer, and they identify two recurring design patterns: retaining raw conversations as retrievable evidence and intent-specific composition ('Query-Routed Scaffolding', RQS). Ablations support both patterns, transplanting RQS into five existing memory systems improves accuracy on all five, cross-backbone and cross-benchmark transfers hold without further evolution, and the accuracy-cost objective shifts computation from build-time LLM extraction to query-time processing.","tokens_in":26648,"tokens_out":3368,"duration_ms":20089,"significance":"If the results hold, the paper is a useful empirical contribution to agent memory design: it gives a concrete framework for whole-program evolution, full prompts and integration details, and it provides falsifiable, checkable claims about two design patterns (raw-conversation retention and query-routed composition) that are tested by ablation and by transplant into five third-party systems. The cost-aware analysis is a genuine addition to a literature that mostly optimizes accuracy alone, and the cross-benchmark/cross-backbone transfer results are a strong, non-obvious check on overfitting. The paper also ships complete prompt templates, documented subset partitions, and code, which raises the reproducibility standard for this type of work. The empirical pattern claims are plausible and useful even if the exact magnitude of the reported gains is partly contingent on the evaluation partitions and on the single evolutionary runs reported.","major_comments":[{"comment":"The final reported program for each setting is selected as the highest-evaluation-set-accuracy candidate among the shortlist, so the 80% evaluation set is used both for selection and for reporting. The paper states this explicitly (Appendix B.2), but it does not discuss how much of the reported margin could reflect selection on this particular evaluation partition. A genuinely unused held-out test split, a multiple-split estimate, or a bias-corrected selection analysis would be needed to establish the magnitude of the headline gains; the cross-benchmark transfer results provide indirect evidence of generalization but do not directly quantify within-benchmark selection bias.","section":"§2.5 and Appendix B.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The LoCoMo search partition contains only two conversations (conv-26 and conv-30, 233 search questions) out of ten. Because both the evolved programs and the reported transfer results are tied to this narrow search set and to the other eight conversations as the evaluation set, the 'up to 12%' headline may be quite sensitive to which two conversations were used for search. The paper should either provide a stability check across alternative splits or explicitly soften the claim from an estimate of expected gain to a result conditional on this fixed split.","section":"Appendix C.1, Table 4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The cross-benchmark transfer comparisons are not fully controlled: on LoCoMo-to-LongMemEval transfer, the evolved programs (65.50% and 64.25%) do not beat RAG (66.00%), although the text in §4.1 says only that they outperform the five memory systems; on LongMemEval-to-LoCoMo transfer, MAPE-ALME's 75.90% exceeds all baselines. The claim 'selected systems outperform the evaluated human-designed baselines' is stated broadly in §1, yet the transfer tables show that this does not hold for RAG in one direction, and the abstract's phrasing could mislead readers into thinking the evolved programs beat every baseline everywhere. The claims in the abstract and introduction should be reworded to match exactly the comparisons that were actually run and won.","section":"§4.1, Tables 1 and 2 and Appendix D.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"LongMemEval evolution is run only once per setting, and each run has 20 steps; the paper notes in Limitations that trajectories can vary across runs but does not report variance or repeated-run stability for the LongMemEval numbers. Given that the selection procedure picks the best evaluation-set candidate from a shortlist, a single run provides no indication of how much the 77.00% and 72.25% results would move under a different random seed or proposer trajectory.","section":"Appendix B.1, §2.4","authors_acknowledge":"Evolutionary trajectories and resulting programs can vary across runs, and program selection and design interpretation involve empirical judgment.","authors_acknowledge_p":0.68}],"minor_comments":[{"comment":"The sentence 'LoCoMo is split by conversation, while LongMemEval-S is split by question instance together with its associated history' should clarify that LoCoMo's split inherits questions from the conversation partition, since Appendix C.1 states this explicitly; a reader of §3 alone might think questions were split directly.","section":"§3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The cost formula is missing an explicit sum over lifecycle stages s that is described in prose; Eq. (2) defines per-stage cost, and the aggregation over stages appears only in the following sentence. Adding the stage index to the displayed sum would remove the ambiguity.","section":"§2.3, Eq. (2)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The estimated evidence completeness c-hat = a + (1-a)r is definitionally tied to accuracy, and the paper acknowledges this; the Spearman correlation between r and accuracy is the more informative quantity, but it is reported only in prose with no confidence interval or sample-size statement. Reporting the correlation with its uncertainty would strengthen the error-attribution claim.","section":"Appendix D.4, Figure 8","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper states that adversarial questions are excluded from evaluation, but it does not explain what adversarial means on LoCoMo or whether the exclusion is standard for the benchmark; one sentence in Appendix C.1 defining the exclusion criterion would remove ambiguity.","section":"Table 4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The ranking reversal between MAPE-ALCM and MAPE-PLCM under Qwen is attributed to MAPE-ALCM's failure to resolve relative dates, but Table 1 shows that MAPE-ALCM also drops on temporal questions while MAPE-PLCM improves; the explanation would be more convincing if the claimed 134-of-145 error pattern from Appendix D.1 were reported with the corresponding total temporal-error count for MAPE-PLCM.","section":"§4.1","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The paper introduces MAPE, a framework that evolves entire memory programs from minimal seeds using an LLM-driven search loop. The main empirical result is that evolved programs outperform several human-designed baselines on two QA benchmarks, often at lower serving cost. The paper also identifies two recurring design patterns—retaining raw conversations and query-routed composition—and shows that transplanting the composition module into existing systems improves accuracy across the board.\n\nWhat is actually new: the combination of evolving all three lifecycle stages (build, retrieve, compose) from near-empty seeds, the joint accuracy–cost objective, and the systematic ablation/transplant analysis that isolates which design choices matter. The paper is well-written, the appendices are thorough, and the code is released. The cross-backbone and cross-benchmark transfer experiments are a genuine strength—they show the patterns generalize beyond the search setting.\n\nThe soft spot flagged by the stress test is real: the evaluation set (80% of data) is used for final program selection among shortlisted candidates. The paper acknowledges this involves empirical judgment but does not discuss overfitting risk. However, the concern is partly contained by (a) the transfer experiments, which test on a different benchmark without further evolution, and (b) the fact that the search set is held disjoint during evolution. A held-out test set would be cleaner, but the paper's design is not unusual for evolutionary search work. I would call this a moderate concern, not a fatal one.\n\nOther soft spots: only one evolutionary run per setting on LongMemEval, no direct comparison with the closest prior work (MemPro, MemScope), and cost estimates use API rates for a locally served model. These are minor.\n\nWho this is for: researchers working on agent memory, LLM-based program synthesis, or automated algorithm design. The paper deserves a serious referee—the empirical work is careful enough to engage with, and the design patterns are worth discussing even if the selection concern means the exact numbers should be taken with a grain of salt.\n\nRecommendation: accept for peer review. The paper advances the subfield and the methodology is reproducible enough to verify.","headline":"A solid empirical paper on evolving agent memory systems; the evaluation-set selection concern is real but the paper's own transfer experiments partly mitigate it.","tokens_in":27176,"tokens_out":522,"duration_ms":58701},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Evolved memory agents beat hand-built memory systems by up to 12 accuracy points at 35% of the serving cost.","keywords":["program evolution","agent memory","LLM agents","long-conversation QA","evolutionary design search","memory design","query-routed scaffolding","multi-objective search"],"falsifier":"Re-run the same evolution from the same seeds on a different pair of LoCoMo conversations and a different shuffled 100-instance LongMemEval search partition, then test the selected programs on a third held-out partition no search touched; if accuracy margins over the strongest baselines shrink toward zero or the two design patterns stop recurring, the reported gains stem from the fixed splits rather than the evolved designs. A component-level check would take a system that already retains raw conversations and applies intent-specific composition and show that adding RQS gives no gain.","tokens_in":29776,"feed_emoji":"🧠","tokens_out":4578,"duration_ms":65206,"temperature":0.7,"pith_summary":"The paper claims that program evolution can replace much of manual memory engineering: starting from a seed program that stores nothing, an LLM-driven propose-and-diagnose loop evolves the entire memory pipeline as executable code, guided by measured accuracy and serving cost. On LoCoMo and LongMemEval, the selected programs outperform every evaluated human-designed baseline on their home benchmark, with gains up to 12 absolute accuracy points at as little as 35% of the serving cost, and the gains survive changing the serving model and moving to the other benchmark. The paper identifies two recurring design lessons: keep raw conversations as retrievable evidence because extracted summaries drop details, and route retrieved evidence through query-intent-specific scaffolds before answering. It also shows that the optimization objective shapes where computation goes, shifting from build-time LLM extraction to selective query-time processing when cost is optimized together with accuracy.","feed_headline":"Evolved memory agents beat hand-built designs at a third of the cost","feed_subtitle":"Whole-program evolution from bare seeds finds memory designs that beat hand-built baselines and transfer across models.","key_machinery":"The argument is carried by MAPE's standardized lifecycle interface, which exposes memory construction, retrieval, composition, and answering as four programmable stages, so the entire pipeline is open to revision under a shared evaluation protocol. Evolution loops through an LLM proposer that writes candidate programs with design hypotheses, a sanity check, evaluation on a fixed 20% search partition, and a diagnosis agent that attributes failures to build, retrieve, compose, or answer using execution traces. The joint objective uses the accuracy-cost Pareto frontier to guide proposals and retain cheap alternatives. The named artifact is Query-Routed Scaffolding (RQS), a compose-stage module that routes each query by intent and applies an intent-specific scaffold, such as date arithmetic for temporal questions or programmatic counting for aggregation, followed by a single deliberation call before the fixed answering model. RQS is validated by ablations and by transplanting it through a fixed adapter into five human-designed memory systems.","core_discovery":"On the paper's own terms, the central claim is that whole-program evolution from minimal seeds yields memory systems that outperform manually designed baselines and reveal transferable design patterns. All four selected systems, one per benchmark and objective, beat the evaluated human-designed baselines on their respective benchmarks, reaching 87.94% accuracy on LoCoMo and 77.00% on LongMemEval, with the joint objective delivering up to 12.01 absolute points over SimpleMem at about 35% of its serving cost. Two patterns recur across selected programs: retaining raw conversation turns as retrievable evidence and composing retrieved evidence through a named Query-Routed Scaffolding module before the fixed QA model answers. Ablations show that removing raw rounds or the scaffold costs several accuracy points on both benchmarks, and transplanting the scaffold into five existing memory systems improves accuracy in all five by 2.75 to 9.25 points. The authors also argue that the search objective acts as a design lever: accuracy-only search stacks operations and inflates cost for marginal gains, while joint accuracy-cost search keeps a branched lineage and ends with designs that compute selectively at query time.","pith_inferences":["The raw-conversation result suggests practitioners can test the cheapest robustness fix first: keep an indexed raw-history side channel alongside extracted summaries, without changing the rest of the pipeline.","The RQS transplant gains being largest on temporal and counting questions implies that post-retrieval composition is the most neglected stage in current memory designs, and the ranking reversal across backbones suggests making temporal robustness an explicit search criterion would improve generalization.","The single-backbone reversal hints that evolution can overfit to one serving model's date-reasoning habits, so evolving under multiple backbones at once would be a natural extension of the same loop.","The cost-objective result implies that build-time reflection pipelines may not be the best use of LLM tokens for long-history conversational QA, and query-time selective computation is a cheap, testable alternative."],"forward_implications":["Manual memory engineering can be partly replaced by objective-guided evolution: the strongest reported results come from programs that were searched, not designed, and they beat every evaluated hand-built baseline on both benchmarks.","The two recurring design lessons, raw-conversation retention and query-intent composition, survive across two benchmarks, two serving backbones, and, via RQS transplantation, five existing memory systems, so they are candidate general principles for conversational memory.","Optimizing cost alongside accuracy relocates computation from build-time LLM extraction to query-time selective processing, cutting serving calls by 40 to 90 percent while retaining competitive accuracy.","Cross-backbone robustness is not automatic: the accuracy-only program loses 36 points on temporal questions when the serving model changes, while the cost-aware program that resolves dates in code keeps its temporal accuracy.","Because every candidate carries a hypothesis, evaluation results, and failure attribution, the discovered designs yield inspectable lessons rather than opaque programs."],"supporting_citations":[{"why":"Supplies the LoCoMo benchmark, its ten conversations, and the question categories on which the evolved programs are measured.","marker":"Maharana et al., 2024"},{"why":"Supplies the LongMemEval-S benchmark with its six question types used as the second search and evaluation ground.","marker":"Wu et al., 2025"},{"why":"Provides the LLM-as-a-judge protocol that scores answer correctness on both benchmarks.","marker":"Zheng et al., 2023"},{"why":"Provides the Mem0 baseline and a transplantation target for the RQS composition module.","marker":"Chhikara et al., 2025"},{"why":"Provides the A-Mem baseline and an additional transplantation target for RQS.","marker":"Xu et al., 2025"},{"why":"Introduces the composition concept that motivates MAPE's programmable compose stage.","marker":"Xu et al., 2026"},{"why":"Provides the RAG baseline, the strongest evaluated system on LongMemEval that the evolved programs must beat.","marker":"Lewis et al., 2020"},{"why":"Provides the Pareto-dominance criterion that defines the joint accuracy-cost search objective.","marker":"Deb et al., 2002"}],"fun_headline_variants":[],"cache_read_input_tokens":0,"weakest_assumption_plain":"The fixed 20/80 search and evaluation splits are representative enough that selecting the program with the highest evaluation-set accuracy does not inflate reported gains: LoCoMo search uses only two conversations (233 questions) and LongMemEval uses 100 of 500 instances, and an unrepresentative split could overstate margins and transfer results.","fun_headline_variants_meta":{"raw":{"variants":[]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":7.7e-05,"raw_usage":{"cost":7.65492e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":7.65492e-05,"upstream_inference_prompt_cost":7.58212e-05,"upstream_inference_completions_cost":7.28e-07},"total_tokens":1042,"prompt_tokens":1038,"completion_tokens":4,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":0}},"tokens_in":1038,"tokens_out":4,"cost_source":"openrouter_usage","duration_ms":2180,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":0,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T10:39:06.895191+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Re-run the same evolution from the same seeds on a different pair of LoCoMo conversations and a different shuffled 100-instance LongMemEval search partition, then test the selected programs on a third held-out partition no search touched; if accuracy margins over the strongest baselines shrink toward zero or the two design patterns stop recurring, the reported gains stem from the fixed splits rather than the evolved designs. A component-level check would take a system that already retains raw conversations and applies intent-specific composition and show that adding RQS gives no gain.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}