{"id":"48464410-acdc-4471-9ab2-73b60a8c89f1","arxiv_id":"2608.11879","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A controlled benchmark shows that memory-system serving cost is not predictable from conversation length and message size, and that break-even against full-transcript serving ranges from immediate to never within 400 turns.","lead":"This paper measures how much three agent memory systems cost to serve as a conversation grows, and compares that with resending the whole transcript each turn. It shows that the cheapest and most accurate systems are not the same, and that the cost depends on the system, the backbone model, and the conversation length.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Leave-one-cell-out failure of the narrow log-log model (Eq. 1) is used to conclude that cost is driven by internal memory state; this inference requires that no (L,t)-only model could fit, which the paper does not establish.","rationale":"The reader's weakest assumption identifies grid sparsity and the risk that the LOOCV result reflects misspecification of the log-log separable form rather than a genuine internal cost driver. I agree, and I would sharpen the point: even a dense grid would not justify the causal conclusion without testing a broader class of (L,t)-only models. The paper's own limitation statement ('Cost model is descriptive, not mechanistic') shows the authors are aware that the regression does not identify the mechanism, yet the abstract's phrasing overstates what the evidence supports. This does not undermine the measured break-even analysis, the matched cost-accuracy tables, or the headline observation that the simple two-predictor model fails; those remain valuable. It does mean the primary contribution should be reframed as a negative result about a specific model class, conditional on the proposed alternative-model check. Because the reader's verdict is already CONDITIONAL, and this concern strengthens the conditions rather than overturning the paper's overall value, I recommend leaving the verdict unchanged.","tokens_in":17129,"tokens_out":6844,"duration_ms":74516,"concrete_test":"Re-run the leave-one-cell-out procedure of Section 3.3 on the same per-turn cost logs with a strictly more general (L,t)-only model: add an interaction term and a cumulative-token threshold indicator at the documented 30k/40k triggers, or fit a GAM with smooth terms in log(L+1) and log(t+1) plus an interaction. If such an (L,t)-only model achieves LOOCV-MAPE comparable to the baselines (<10%) for the memory systems, the abstract's claim that cost 'cannot be predicted from conversation length and message size alone' is falsified, and the conclusion should be weakened to 'the separable log-log model fails.' If the alternative model still misses by 18–69%, the concern is resolved and the internal-state interpretation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central inference is that the memory systems' high leave-one-cell-out error (18–69%, Table 1) establishes the abstract's claim that serving cost 'cannot be predicted from conversation length and message size alone' and is 'driven instead by internal memory behavior.' This inference is load-bearing and unsupported because the only model tested is the separable log-log form log(C+1)=a+p·log(L+1)+q·log(t+1) (Eq. 1), fitted on five cells for Mem0/Hindsight and seven for Mastra OM (Appendix B). A high LOOCV-MAPE for this narrow class is consistent with any cost function that is a smooth or nonlinear function of (L,t) — for example, threshold-triggered consolidation at cumulative-token budgets (Mastra OM fires the observer at 30,000 and the reflector at 40,000 accumulated tokens, Table 5) — without any 'internal memory state' beyond a deterministic function of depth and message size. The paper itself concedes the model is 'descriptive, not mechanistic' (Section 5.4), which undercuts the causal reading in the abstract. Thus the main negative result is currently 'a separable power-law in L and t fails to predict,' not 'no model using only conversation length and message size can predict.' The sparse grid compounds the problem: with only 4–6 nontrivial folds for the memory systems, one atypical corner cell can dominate the MAPE and masquerade as a mechanistic signature.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks the serving cost of three agentic memory systems (Mem0, Hindsight, Mastra Observational Memory) against two reference strategies (a 10-turn rolling window and full-transcript resubmission), across two backbones at two reasoning-effort settings, using synthetic conversations up to 400 turns. Each cost measurement is paired with accuracy on a 665-question LoCoMo subset. The authors fit a separable log-log model log(C+1)=a+p·log(L+1)+q·log(t+1), report leave-one-cell-out MAPE, compute break-even lengths from measured per-turn costs, and assemble a joint cost-per-correct-answer matrix. The main findings are that the cost model generalizes well for the two baselines (LOOCV-MAPE 2.9–6.5%) but poorly for the memory systems (18–69%), that break-even timing is highly workload- and backbone-dependent, and that no single system dominates on both cost and accuracy.","tokens_in":17400,"tokens_out":4179,"duration_ms":43822,"significance":"If the benchmark results hold, the paper provides a genuinely useful controlled measurement of memory-system serving cost: it uses external systems and an external QA corpus, pairs cost and accuracy in matched configurations, caches identical dialogues across systems, uses a cluster bootstrap that respects run-level dependence, and reports failures transparently (Hindsight's uncontrolled ingest, the gpt-oss-20b serving-stack rejection at turn 374, and provider-routing variance). The fitted exponent separation for the reference baselines (p≈q≈1 for full history; q≈0.1 for the rolling window) is a clean, interpretable validation of the measurement methodology. The main weakness is that the central negative claim is stated more strongly than the evidence supports: the paper tests one simple parametric model, not the general proposition that cost cannot be predicted from conversation length and message size. The benchmark is still a solid contribution and a useful target for future, more mechanistic cost models, provided the claims are recalibrated.","major_comments":[{"comment":"The central claim that a memory system's serving cost 'cannot be predicted from conversation length and message size alone' and is 'driven instead by internal memory behavior' is supported only against the specific separable power-law model in Eq. (1). An 18–69% LOOCV-MAPE shows that this one functional form fails; it does not establish that no (L,t)-only model can predict. In fact, Mastra OM's documented thresholds (observer at 30,000 accumulated message tokens, reflector at 40,000 accumulated observation tokens, Table 5) are deterministic functions of conversation length and message size, so a piecewise or cumulative-token specification would plausibly track its cost. The manuscript's own Section 5.4 concedes the model is 'descriptive, not mechanistic,' which undercuts the causal phrasing in the abstract. I recommend reframing the claim to 'a simple separable power law fails, and cost appears to depend on internal pipeline events such as threshold-triggered consolidation,' and testing at least one alternative (L,t)-only specification, such as a model with cumulative-token terms or threshold indicators.","section":"Section 4.1, Eq. (1), Table 1, and Abstract"},{"comment":"With five cells for Mem0 and Hindsight and seven for Mastra OM, leave-one-cell-out MAPE can be dominated by a single atypical corner. The per-setting max APE for Mastra OM reaches 1.918 (Table 6), meaning one held-out cell is off by nearly 2×; averaging this with the other folds produces the reported 0.685. The statement that held-out failures 'cluster at the grid corners' needs a per-fold error table, and the grid should be densified around the 30k/40k token thresholds to test whether a cumulative-token model removes the apparent internal-state dependence. As written, the comparison between baseline and memory-system LOOCV error partly confounds model misspecification with grid sparsity.","section":"Appendix B and Table 6"},{"comment":"Hindsight's ingest stage ran under a single uncontrolled configuration for all settings, yet Hindsight's dollar costs and break-even turns appear in the headline joint tables and in the cost-per-correct-answer ranking. On the Gemma 4 26B A4B columns, for example, Hindsight's ingest tokens were billed at gpt-oss-20b rates, so its $0.051–$0.052 cost is not commensurable with Mem0 and Mastra OM costs on the same column. This contaminates the 'no system wins on both axes' conclusion and the cross-system break-even comparisons. The paper should either re-run Hindsight with controlled ingest or present Hindsight entries in a clearly separated 'uncontrolled' panel, with explicit statements that its dollar figures are not directly comparable to the other systems'.","section":"Section 5.4 and Tables 3, 4, 8"}],"minor_comments":[{"comment":"The abstract has missing spaces in 'by18–69%' and 'by21–54%'; please fix the typesetting.","section":"General / Abstract"},{"comment":"The paper states confidence intervals for p and q come from a cluster bootstrap, but Table 6 reports only point estimates. Please report the intervals, or state where they are available (e.g., in the supplementary material).","section":"Section 3.3"},{"comment":"The '+1' inside the logarithms is not justified in the text. Since costs can be small at early turns, the addition may matter for the fit; please state the convention and its units.","section":"Section 3.3, Eq. (1)"},{"comment":"Figure 2 extends the full-history curve with a fitted dashed line beyond turn 374, while the text emphasizes that break-even is computed from measured costs with no fitted-model extrapolation. Clarify that the dashed extension is illustrative and does not enter any break-even computation.","section":"Section 4.2 and Figure 2"},{"comment":"The explanation that higher reasoning effort 'uses up the max_tokens budget and leaves less room for the answer' is plausible but not directly supported by reported diagnostics. If kept, report the relevant max_tokens consumption or output-token counts for the affected cells.","section":"Section 5.2"},{"comment":"Accuracy is judged by a single LLM judge with a single pass at temperature 0. Reporting judge agreement or a small sensitivity analysis would strengthen the accuracy comparisons, especially for the smaller differences between systems.","section":"Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the benchmark design is genuinely useful and the authors are unusually transparent about their failure modes, which should be preserved. The main revision should focus on recalibrating the central causal claim and adding at least one alternative cost model; I would not reject the paper, but I would not accept it while the abstract asserts 'cannot be predicted' and 'driven by internal memory behavior' on the basis of a single parametric family's LOOCV failure. Please also encourage the authors to release the harness and per-fold error data, since the paper's value as a benchmark depends heavily on reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful thing: this is the first controlled measurement of serving cost across three agentic memory systems against two reference strategies, with accuracy on the same runs. The break-even analysis is genuinely new and is the part I trust most: it uses measured per-turn costs, no fitted extrapolation, and it quantifies what practitioners actually need to know — when a memory system pays for itself, and that under some backbones it never does within 400 turns. The honest reporting is also a plus: they disclose Hindsight's uncontrolled ingest stage, the OpenRouter routing variance, and the serving-stack rejection at turn 374.\n\nThe soft spot is the abstract's headline claim that serving cost 'cannot be predicted from conversation length and message size alone' and is 'driven by internal memory behavior.' That conclusion rests on a leave-one-cell-out test of one narrow model — a separable power law — on five to seven cells. High LOOCV error for that family doesn't rule out, say, a piecewise model with a threshold on cumulative tokens. Mastra OM's observer fires at 30k and reflector at 40k accumulated tokens; those are deterministic functions of (L,t), so a model with that structure might fit fine. The paper even concedes the model is 'descriptive, not mechanistic' in Section 5.4, which undercuts the causal reading in the abstract. The claim should be softened to 'a simple separable power law fails,' or the authors should test a wider class of models.\n\nOther concerns are minor: no code or data release yet, Hindsight's ingest is not benchmark-controlled (flagged but limits cross-system comparisons), accuracy rests on one subset of LoCoMo with dialogue-level clustering caveats they acknowledge. None of these break the break-even or cost-accuracy findings.\n\nWho's this for? Practitioners choosing a memory system and people building cost benchmarks. It's a solid data point, not a final verdict.\n\nI'd send it to peer review, conditional on the authors fixing that overclaim and releasing artifacts. It's worth referee time.","headline":"Break-even data and honest reporting make this worth reading, but the 'unpredictable' claim outruns the evidence; the abstract needs softening.","tokens_in":17950,"tokens_out":3872,"would_cite":false,"duration_ms":39088,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The per-turn serving cost of agentic memory systems is driven by internal pipeline state, not by conversation length or message size.","keywords":["agentic memory systems","LLM serving cost","cost benchmarking","break-even analysis","cost-accuracy trade-off","LoCoMo","conversational memory","cost modeling"],"falsifier":"Run the same three memory systems on a denser grid, say twenty or more (N,L) cells; if a pure length-and-size power-law model then holds leave-one-out error below roughly 7% for Mem0, Hindsight, and Mastra Observational Memory, the claim that internal state drives cost would be refuted. The paper's own baseline error range sets the bar: 2.9–6.5% for the two reference strategies.","tokens_in":16900,"feed_emoji":"🧠","tokens_out":6771,"duration_ms":56762,"temperature":0.7,"pith_summary":"Long-running chatbots use memory systems to avoid resending the whole conversation, but the billable cost of those systems has not been systematically measured. This paper benchmarks three memory systems against a fixed rolling window and full-transcript resubmission across two backbones, and pairs every cost measurement with accuracy on 665 LoCoMo questions. Its central finding is that per-turn serving cost for memory systems cannot be predicted from conversation length and message size: a model that fits the two reference strategies within 2.9–6.5% held-out error misses the memory systems by 18–69%, because their cost is set by internal extraction, retrieval, and consolidation behavior. It also finds that the conversation length at which a memory system becomes cheaper than resubmitting the transcript ranges from turn 0 to never within 400 turns, and that no system is both cheapest and most accurate. If right, the result tells practitioners that memory-system cost is a property of the pipeline, not of the conversation, and that the choice of backbone matters as much as the choice of memory system.","feed_headline":"Serving cost of AI memory follows internal state, not chat length","feed_subtitle":"Three memory systems miss length-and-size cost models by 18-69%; break-even timing ranges from turn 0 to never.","key_machinery":"The central instrument is the two-term power-law cost model $\\log(C+1)=a+p\\log(L+1)+q\\log(t+1)$ with one exponent for message size and one for conversation depth, validated by leave-one-cell-out cross-validation. The model is designed to separate two cases that a single cumulative-content predictor treats as identical: a long conversation with small messages and a short conversation with large messages. It carries the argument by showing that the baselines' exponents match their known mechanisms while the memory systems' exponents do not generalize to held-out cells; the held-out error, not the in-sample $R^2$, is the diagnostic that exposes internal memory behavior. The break-even length, defined as the turn at which a system's cumulative cost falls below full-history cost and stays below, turns the fitted and measured costs into an actionable deployment quantity.","core_discovery":"The paper claims that a memory system's serving cost is governed by its internal memory state rather than by the size of the conversation being served. Evidence comes from a separable log-log cost model $\\log(C+1)=a+p\\log(L+1)+q\\log(t+1)$ fitted per system: the full-history baseline fits $p\\approx q\\approx 1$, the rolling window fits $q\\approx 0.1$, and both hold out below 6.5% error, while Mem0, Hindsight, and Mastra Observational Memory hold out with 18–22%, 46–48%, and 41–69% error respectively. The paper reads the high held-out error as a positive finding: internal triggers such as fact extraction, retrieval payload, and consolidation thresholds dominate the bill, so external workload descriptors cannot predict cost. It further claims that break-even against full-history resubmission is workload-dependent, and that accuracy spans 21–54% with no system winning on both cost and accuracy.","pith_inferences":["A mechanistic cost model that tracks per-stage internal variables—number of extracted facts, retrieval payload size, and consolidation triggers—should be able to predict memory-system serving cost with held-out error closer to the baselines; the paper names this as future work.","If the internal-state result holds, published memory-system cost comparisons are tied to the backbone they were measured on, so a system that looks cheap on one model may not be cheap on another.","The break-even framework suggests a practical deployment rule: estimate expected conversation length in turns, compare it with a system's break-even length on the chosen backbone, and keep full-transcript resubmission as the default below that threshold.","A natural test is to rerun the same protocol on a task-oriented dialogue corpus, such as MultiWOZ 2.2, to see whether the internal-state cost driver appears outside persona-grounded chat."],"forward_implications":["When the full transcript is resubmitted every turn, cost scales nearly linearly in both message size and depth ($p\\approx q\\approx 1$), so as conversations grow long a memory system is eventually cheaper—but within 400 turns Hindsight can still be costlier than the transcript.","Mastra Observational Memory can break even at turn 0, Mem0 within tens of turns for large messages, and Hindsight sometimes never, so the expected conversation length decides whether a memory system is economical.","The backbone choice shifts cost as much as the memory system choice: Mem0's 100-turn cost drops from $\\$0.059$–$\\$0.065$ on gpt-oss-20b to $\\$0.019$ on Gemma 4 26B A4B, and raising reasoning effort can lower accuracy (Mem0 drops from 0.322 to 0.214).","No system wins on both axes; the cheapest per correct answer is Mastra Observational Memory on gpt-oss-20b low in one cell and Mem0 on Gemma 4 26B A4B in another, while Hindsight is the most expensive at the 100-turn reference cell.","Cost predictions for the three memory systems at unseen conversation lengths should be treated as rough estimates until a mechanistic model of their internal state exists."],"supporting_citations":[{"why":"supplies Mem0, one of the three benchmarked memory systems, with its flat extract-and-retrieve pipeline.","marker":"[6]"},{"why":"supplies Hindsight, the retain-recall-reflect memory system whose ingest overhead makes it the most expensive at the reference cell.","marker":"[14]"},{"why":"supplies Mastra Observational Memory, the threshold-triggered observer-reflector-actor system whose cost is set by internal triggers.","marker":"[3]"},{"why":"supplies LoCoMo, the multi-session dialogue dataset and the 665 evaluated questions for the matched accuracy measurements.","marker":"[17]"},{"why":"supplies gpt-oss-20b as one serving backbone and gpt-oss-120b as the fixed judge model.","marker":"[19]"},{"why":"supplies Gemma 4 26B A4B as the second serving backbone and the dialogue generator.","marker":"[10]"},{"why":"supplies leave-one-out cross-validation, the diagnostic that separates the generalizing baselines from the memory systems.","marker":"[26]"}],"fun_headline_variants":["AI memory cost taps internal state, not chat size","Memory systems' bill driven by internals, not context","Serving cost of agent memory defies length-based models","Agent memory pricing: internal triggers beat conversation size"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's conclusion that memory-system cost is driven by internal state rests on the assumption that the few sampled (conversation-length, message-size) cells are representative enough for leave-one-cell-out error to signal a real cost driver, and on the benchmark controlling each system's internal stages—an assumption that fails for Hindsight, whose ingest stage ran under a fixed configuration the benchmark did not control.","fun_headline_variants_meta":{"raw":{"variants":["AI memory cost taps internal state, not chat size","Memory systems' bill driven by internals, not context","Serving cost of agent memory defies length-based models","Agent memory pricing: internal triggers beat conversation size"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1312,"prompt_tokens":956,"completion_tokens":356,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":292}},"tokens_in":572,"tokens_out":356,"duration_ms":3928,"temperature":1.0,"reasoning_tokens":292,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:23:32.329314+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three memory systems on a denser grid, say twenty or more (N,L) cells; if a pure length-and-size power-law model then holds leave-one-out error below roughly 7% for Mem0, Hindsight, and Mastra Observational Memory, the claim that internal state drives cost would be refuted. The paper's own baseline error range sets the bar: 2.9–6.5% for the two reference strategies.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies Mastra Observational Memory, the threshold-triggered observer-reflector-actor system whose cost is set by internal triggers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies leave-one-out cross-validation, the diagnostic that separates the generalizing baselines from the memory systems."}],"review_version":1}