{"id":"14846ea1-618a-4f29-9332-7911d4a888d0","arxiv_id":"2608.04843","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A trainable memory system that distills memory construction into a small model and uses a GRPO-trained summarizer to improve answer quality per unit inference cost in long-horizon agent conversations.","lead":"This paper introduces MemoryCPT, a two-stage trainable memory pipeline for LLM agents that learns to build memories offline and to retrieve and compress them online under a cost-aware reward. The authors report better answer quality per inference dollar than several baselines on two long-conversation benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cost accounting for offline memory construction is internally inconsistent; if the LoCoMo test split uses two stores, corrected QPC falls below Memory-R1 and the headline claim flips.","rationale":"The reader correctly identified the cost model as the weakest point, but the most load-bearing defect is not the exclusion of adapter training; it is the internal inconsistency in the amortized offline cost. The paper's own appendix numbers do not reproduce the main-table Cost or QPC, and the paper uses at least two different values for the same quantity (3.46 in the narrative vs 4.31 in Tables 2/3, and 0.138 vs 0.111 for QPC). Because the margin over Memory-R1 on LoCoMo is small, a modest upward correction to the amortized offline term flips the primary comparison. This is a concrete, reparable accounting issue, not a matter of differing community standards, so a conditional verdict is appropriate: the authors should clarify the store count and recompute the cost tables. If the corrected numbers confirm the flip, the LoCoMo-specific central claim would fail; if the correction is small, the qualitative conclusion may survive. The concern is fair to the paper in that it is based on the authors' own stated definitions and table values, and it can be settled by a straightforward arithmetic check.","tokens_in":16597,"tokens_out":14868,"duration_ms":163049,"concrete_test":"Ask the authors to state how many memory stores were built for the LoCoMo test split and to release per-store QAD token counts. If conv-49 and conv-50 are two separate stores serving 314 questions, recompute Table 2/3 LoCoMo Cost as C_online + C_QAD*(1/n1 + 1/n2), with n1 + n2 = 314, and QPC = F1/Cost. If the corrected total equals or exceeds 4.87, MemoryCPT's QPC drops below Memory-R1's 0.098 on LoCoMo, invalidating the 'best cost-performance' claim on that benchmark; if only one store was built, or if the reported 4.31 already includes all test-store construction costs, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim rests on a cost model that is not arithmetically reproducible. Equation 3 amortizes C_QAD over n queries; the appendix defines C_QAD as the offline construction cost per complete memory store, reports C_QAD = 0.02542 USD for Qwen2.5-7B (Tables 7/9), and reports C_online = 3.462e-4 USD per query. For the stated LoCoMo split of conv-49 and conv-50 with 314 test questions, the appendix's own components give Cost = 3.462e-4 + 0.02542/314 = 4.272e-4 (i.e., 4.27 in cost units), not the 4.31 reported in Tables 2 and 3. The 4.31 value instead matches amortizing C_QAD over roughly 300 queries, and the narrative value 3.46 is C_online without amortization. More importantly, if the two test conversations require two memory stores, the correct amortized offline term is at least 2*0.02542/314 = 1.62e-4, giving Cost ≈ 5.08 and QPC = 0.479/5.08 ≈ 0.094, below Memory-R1's LoCoMo QPC of 0.098. The reported QPC advantage over Memory-R1 is only 13% (0.111 vs 0.098), so this cost-correction is large enough to reverse the primary LoCoMo ranking. This is more directly load-bearing than the excluded adapter-training cost, which the paper explicitly scopes out as inference cost and which depends on an arbitrary amortization horizon.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MemoryCPT, an end-to-end trainable memory framework for long-horizon LLM agents, consisting of two stages: Query-agnostic Distillation (QAD), which distills a modular memory-construction pipeline into a compact model via reasoning-trace SFT, and Query-aware Retrieval and Summarization (QAR), which uses RRF-based retrieval plus a GRPO-trained LoRA summarizer with a cost-aware reward. The paper introduces Quality per Cost (QPC) as an evaluation metric and reports experiments on LoCoMo and LongMemEval showing improved cost-performance trade-offs over baselines such as Memory-R1, BudgetMem, LightMem, and MemoryOS. The appendix provides detailed token and cost breakdowns, hyperparameters, and per-category results.","tokens_in":16964,"tokens_out":6989,"duration_ms":63422,"significance":"If the empirical claims survive correction, MemoryCPT is a meaningful step toward trainable memory pipelines that explicitly optimize the quality-cost trade-off rather than accuracy alone. The two-stage design (offline distillation of memory construction, online query-conditioned summarization) is well motivated, and the QPC metric plus the detailed cost accounting in the appendix are useful contributions to the evaluation methodology. The paper also provides unusually explicit implementation details (token counts, pricing tables, hyperparameters, per-category breakdowns), which is a strength for reproducibility. However, the current numerical inconsistencies between narrative text and tables, the unresolved cost-accounting ambiguity for multiple memory stores, and the test-set-based hyperparameter selection prevent the central claim from being accepted as stated.","major_comments":[{"comment":"The reported LoCoMo cost of 4.31 (in USD×10^4 units) is not reproducible from the stated components. Eq. (3) with C_QAD=0.02542, C_online=3.462e-4, and n=314 gives 3.462e-4 + 0.02542/314 = 4.272e-4, not 4.31e-4; the value 4.31e-4 corresponds to n≈300. More seriously, the test split uses two conversations (conv-49 and conv-50) while Table 7 defines C_QAD as the offline cost per complete memory store. If each conversation requires its own store, the amortized offline term is 2×0.02542/314 = 1.62e-4, yielding Cost≈5.08 and QPC = 0.479/5.08 ≈ 0.094, which is below Memory-R1's LoCoMo QPC of 0.098 in Table 2. The headline advantage over Memory-R1 therefore depends on an unstated store-per-conversation assumption and on the choice of n. Please clarify the number of memory stores for the LoCoMo test set and recompute all costs and QPC values consistently.","section":"Cost and Quality per Cost / Appendix 'Amortized Cost Model'"},{"comment":"The text accompanying Table 2 states that MemoryCPT reduces Cost from 24.11 to 3.46 and achieves QPC 0.138, but the table reports Cost 4.31 and QPC 0.111 for the same configuration. The claims of '7 times lower inference cost' and 'over 9 times higher QPC' are computed from the narrative values (3.46 and 0.138), not from the table values (4.31 and 0.111, which give 5.6× lower cost and 7.4× higher QPC). These discrepancies appear throughout the results section and must be resolved; the reported magnitudes of the claimed advantage are not the same in the text and the tables.","section":"Model Comparison (Table 2)"},{"comment":"The QAD ablation paragraph claims that removing QAD decreases QPC from 0.138 to 0.131, implying QAD improves cost-performance. Table 3 shows the opposite: the w/o QAD row has LoCoMo QPC 0.131, while the full MemoryCPT row has QPC 0.111, so removing QAD increases QPC. The text's conclusion that 'QAD acts as a quality amplifier' is thus not supported by the reported table; it relies on the non-table QPC value 0.138. This is a load-bearing contradiction for the component-level claim and should be corrected and discussed.","section":"Ablation Study (Table 3)"},{"comment":"The default hyperparameters α=0.8 and retrieval depth 20/50 appear to be selected by sweeping on the test sets: Fig. 2 reports that QPC(QA) is maximized at α=0.8 on the LoCoMo and LongMemEval test benchmarks, and Fig. 3 reports the same for the retrieval depth. Because the main results in Tables 2 and 3 use these defaults, the reported performance is partly a consequence of test-set fitting. The paper should move hyperparameter selection to a validation split, or explicitly report the selection procedure on training/validation data and quantify how the main conclusions change under this protocol.","section":"Cost-Performance Trade-off Analytics / Retrieval Depth Analytics (Figs. 2 and 3)"},{"comment":"The cost model amortizes C_QAD, the offline memory construction cost, but does not include the cost of training LoRA-A and LoRA-B, which are retrained for each dataset (Table 4 in the appendix; QAD training sets of 1185 and 4766 examples for LoCoMo and LongMemEval). The title and abstract frame the contribution as a cost-performance trade-off, so excluding training costs should be explicitly justified and, ideally, quantified (e.g., GPU-hours or USD and the amortized contribution per query). Without this, the QPC comparisons are not a complete cost accounting, and the '7× lower cost' narrative is overstated relative to total system cost.","section":"Eq. (3) and Appendix 'Offline Distillation Usage Details'"}],"minor_comments":[{"comment":"The phrase 'an QPC of 0.138' should read 'a QPC of 0.138'.","section":"Model Comparison"},{"comment":"The text uses 'QID semantic-abstraction stage' where the rest of the paper uses QAD (Query-agnostic Distillation); this acronym is undefined and should be fixed.","section":"Sub-Category Performance on LongMemEval"},{"comment":"The cost unit 'USD×104' should be typeset as 'USD×10^4' for clarity.","section":"Table 2 and Table 3 captions"},{"comment":"The quantitative statements in the text (e.g., QPC(QA) peaks of 59.5 and 56.5; retrieval depth knee recovering 96% of peak F1 at 42–44% of candidate cost) refer to figures that are not present in the provided manuscript text; please ensure the figures are included and legible in the final version.","section":"Figures 2 and 3"},{"comment":"All tables report a single run; given the marginal QPC difference between MemoryCPT (0.111) and Memory-R1 (0.098) on LoCoMo, bootstrap confidence intervals or multiple-seed runs would strengthen the ranking claims.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The main concern is that the central cost-performance claim is not yet arithmetically stable: the text and tables disagree on the headline cost and QPC, and the number of memory stores for the LoCoMo test split can flip the ranking against Memory-R1. The test-set hyperparameter selection is also a serious methodological issue that the authors should address rather than paper over. The paper is publishable in principle, but these points need to be fixed before a decision for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely new two-stage memory pipeline, and the QPC framing is useful. But the cost accounting is not reproducible, and when you correct it, the main QPC advantage over Memory-R1 on LoCoMo appears to flip.\n\nWhat is new: QAD distills memory construction with reasoning traces rather than final summaries only, and QAR trains a LoRA summarizer with GRPO under a cost-aware reward. That specific combination is not in the cited baselines, and the two-stage offline/online separation is a sensible decomposition. The paper also ships a lot of detail: token and price tables, per-category results, and a clear ablation structure.\n\nThe soft spots are real and load-bearing. The narrative text and the tables disagree on the central numbers: the text says Cost 3.46 and QPC 0.138, while Table 2 reports 4.31 and 0.111. The QAD ablation paragraph claims QPC falls on LoCoMo, but Table 3 shows it rises from 0.111 to 0.131, which directly undermines the claim that QAD improves cost-performance. These are not rounding differences.\n\nDeeper problem: the amortization. Equation 3 is C_amortized = C_QAD/n + C_online. The appendix reports C_QAD = 0.02542 per memory store and C_online = 3.462e-4. The LoCoMo test split uses two conversations, conv-49 and conv-50, which means two separate memory stores. With n=314 total queries, the correct offline term is 2*0.02542/314 = 1.62e-4, not 0.81e-4. That gives total cost about 5.08 in the paper's units, not 4.31, and QPC about 0.094, below Memory-R1's 0.098. The table’s 4.31 only works if you assume a single store and n≈300. So the primary LoCoMo ranking over the closest RL-based baseline flips under the paper's own cost model.\n\nThere is also a circularity issue: the default reward coefficient alpha=0.8 and retrieval depth 20/50 were selected by sweeping on the test sets themselves, with no validation split displayed. No error bars, single seed. These are reparable, but the current empirical claims need a full re-run under a corrected cost model.\n\nWho this is for: people building practical long-horizon agent memory systems and anyone trying to evaluate memory methods under a unified cost metric. The QPC metric itself is a good addition to the evaluation toolbox, and the method is well-motivated.\n\nRecommendation: send it to peer review, but with a referee who will check every arithmetic step. The current version should not be accepted as-is. If the authors correct the store count, fix the text/table mismatches, and move model selection off the test set, the paper could become a solid contribution.","headline":"The pipeline idea is fresh and the writing is clear, but the cost accounting is wrong enough to flip the headline QPC comparison; needs arithmetic fixes before the empirical claims can be trusted.","tokens_in":17447,"tokens_out":5530,"would_cite":false,"duration_ms":53213,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MemoryCPT claims a trainable memory pipeline that raises answer quality per dollar.","keywords":["agent memory","long-horizon dialogue","cost-performance trade-off","query-aware summarization","reciprocal rank fusion","GRPO","LoRA distillation","Quality per Cost"],"falsifier":"Recompute QPC after adding the measured cost of training LoRA-A and LoRA-B, divided by the number of test queries, to $C_{\\text{amortized}}$ on LoCoMo. If the corrected QPC falls below Memory-R1's 0.098 or below the BudgetMem value, the central cost-performance claim fails under that accounting.","tokens_in":16424,"feed_emoji":"🧠","tokens_out":7395,"duration_ms":69335,"temperature":0.7,"pith_summary":"This paper proposes MemoryCPT, an agent memory pipeline trained end to end rather than assembled from hand-crafted heuristics. It claims that splitting memory into an offline query-agnostic construction stage and an online query-aware retrieval-summarization stage lets a frozen QA model answer long-horizon dialogue questions more accurately while spending fewer tokens. To make the trade-off explicit, the paper introduces Quality per Cost (QPC), the ratio of token-level F1 to amortized inference cost, and reports that MemoryCPT improves QPC over five baselines on the LoCoMo and LongMemEval benchmarks.","feed_headline":"MemoryCPT yields more answers per dollar than five memory baselines","feed_subtitle":"A two-stage trained memory pipeline cuts inference cost sharply while improving F1 on LoCoMo and LongMemEval.","key_machinery":"The carrying mechanism is the two-stage split of $f_{\\text{mem}}$ into $f_{\\text{QAD}}$ and $f_{\\text{QAR}}$. QAD distills a modular teacher pipeline (segmenter, episode generator, episode merger, semantic extractor) into a LoRA adapter via supervised fine-tuning on reasoning traces and structured JSON, then merges it into the base model to build reusable episodic and semantic memory stores offline. QAR retrieves candidates by reciprocal rank fusion over dense and sparse retrievers, then uses a second LoRA adapter trained with GRPO to generate a query-conditioned summary under the reward $\\alpha\\,\\mathrm{F1}(\\hat{y}, y^*) + (1-\\alpha)(1-\\tilde{C}(q,m_q))$, where $\\tilde{C}$ is a clipped, normalized token-cost proxy. Quality per Cost, defined as $\\mathrm{QPC} = \\mathrm{F1}/C_{\\text{amortized}}$, is the evaluation identity that converts both stages into a single number.","core_discovery":"The paper's central claim is that cost and answer quality do not have to be traded off at the system level: a memory pipeline can be trained to retain answer-critical evidence while suppressing redundant tokens. On LoCoMo with a Qwen2.5-7B memory model, MemoryCPT reports F1 0.479 at Cost 4.31 (USD $\\times 10^4$) against BudgetMem's F1 0.373 at Cost 24.11, lifting QPC from 0.015 to 0.111, and it reports a similar pattern on LongMemEval. The same advantage holds with a Llama-3.2-3B base. Ablations attribute the gain to both stages: removing QAD lowers F1, while removing QAR or the fine summarizer raises cost several-fold, so the full pipeline is what lands simultaneously at high F1 and low cost.","pith_inferences":["Editorial inference: if the LoRA-A and LoRA-B training runs were amortized into the cost denominator, the reported QPC advantage would shrink; the paper counts only offline memory construction cost, not adapter training cost.","Editorial inference: the same offline-store/online-compressor split is a transferable recipe for other retrieval-heavy settings, such as document or email memory, where a reusable index can be built once and a small policy learned to summarize under a token budget.","Editorial inference: replacing token-level F1 in the reward with a judge-based or semantically sensitive quality signal would test whether the cost savings persist when correctness is judged by meaning rather than lexical overlap."],"forward_implications":["Memory construction can be amortized: one offline pass builds a store that serves many queries, so per-query cost approaches the online lower bound as interactions accumulate.","Query-aware compression, not just retrieval, is the main cost lever: removing QAR raises LoCoMo cost from 4.31 to 11.10 while lowering F1.","A frozen downstream QA model plus a small trained summarizer can deliver near-best accuracy at a fraction of the inference cost of heuristic pipelines.","Agent memory evaluation should report quality per cost alongside raw accuracy, because comparable F1 scores across systems hide large differences in token spending.","The reward coefficient $\\alpha=0.8$ sits near the Pareto frontier: extreme quality-only or cost-only rewards both degrade the F1-per-dollar result."],"supporting_citations":[{"why":"Supplies the modular four-role teacher pipeline that QAD distills into the student model.","marker":"Ma et al. 2026"},{"why":"Contributes the LoCoMo benchmark and the token-level F1 metric used in the evaluation.","marker":"Maharana et al. 2024"},{"why":"Provides the LongMemEval benchmark and its six memory categories used for the second evaluation set.","marker":"Wu et al. 2025"},{"why":"Provides the budget-aware baseline that MemoryCPT is compared against.","marker":"Zhang et al. 2026"},{"why":"Supplies GRPO, the policy optimization algorithm used to train the LoRA-B summarizer.","marker":"Shao et al. 2024"},{"why":"Defines reciprocal rank fusion, the retrieval-combination method used in QAR.","marker":"Rackauckas 2024"}],"fun_headline_variants":["MemoryCPT boosts answer quality per dollar sevenfold","MemoryCPT cuts inference cost 5.6x, lifts F1","MemoryCPT beats budget baselines on F1 and cost","MemoryCPT: lower cost, higher F1 on long-horizon tasks","Trainable memory distillation: F1 up, cost down"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The cost advantage assumes adapter training costs should not be charged to the queries being answered; if the LoRA-A and LoRA-B training runs are amortized over the 314 test questions, the reported QPC gap narrows.","fun_headline_variants_meta":{"raw":{"variants":["MemoryCPT boosts answer quality per dollar sevenfold","MemoryCPT cuts inference cost 5.6x, lifts F1","MemoryCPT beats budget baselines on F1 and cost","MemoryCPT: lower cost, higher F1 on long-horizon tasks","Trainable memory distillation: F1 up, cost down"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000617,"raw_usage":{"total_tokens":2856,"prompt_tokens":932,"completion_tokens":1924,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":1844}},"tokens_in":548,"tokens_out":1924,"duration_ms":15547,"temperature":1.0,"reasoning_tokens":1844,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:12:49.024613+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute QPC after adding the measured cost of training LoRA-A and LoRA-B, divided by the number of test queries, to $C_{\\text{amortized}}$ on LoCoMo. If the corrected QPC falls below Memory-R1's 0.098 or below the BudgetMem value, the central cost-performance claim fails under that accounting.","supporting_citations":[],"review_version":1}