REVIEW 3 major objections 6 minor 18 references
Store latency alone decides whether an agent can use memory on every reasoning step, and that changes what the agent actually does.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-11 03:43 UTC pith:WMPDCHI5
load-bearing objection Clean causal isolation of store latency under a fixed budget, with honest baselines and real microsecond measurements; the philosophy is optional packaging. the 3 major comments →
Memory in the Loop: In-Process Retrieval as ExtendedWorking Memory for Language Agents
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Holding a fixed per-turn memory-latency budget and varying only the store's answer speed, redundant actions rise monotonically with latency: 0.0 of 12 at in-process speed versus 7.2 of 12 at a 110 ms cloud round trip. Store latency alone therefore determines whether a per-step loop guard can exist and thereby changes task outcomes. At in-process speeds (~100 µs), memory moves inside the agent loop and becomes eligible as extended working memory; network stores force rationing to once per turn.
What carries the argument
Memory in the loop: per-step read/write of an in-process associative store, gated by a feasibility frontier that prices affordable retrieval frequency as f_max = (β/(1-β)) · (r/c). The causal instrument is a fixed per-turn memory budget B under which only store answer speed is varied, so latency changes whether a lookup fits rather than what it returns.
Load-bearing premise
That a designer-chosen fixed memory budget per turn (about 100 ms, or 10% of a one-second step) is the right way to expose the causal path, and that a semantic loop-guard task under a short window stands in for the broader class of per-step memory uses the paper claims become practical once latency collapses.
What would settle it
Rerun the real-LLM loop-guard experiment with identical models, prompts, and seeded workloads, but replace the injected-delay ladder with a live cloud vector store at measured ~110 ms RTT under the same 100 ms per-turn budget: if redundant investigations no longer rise to ~7.2 of 12 and the in-process arm no longer sits at zero, the causal claim fails.
If this is right
- Retrieval frequency (per-turn vs per-step) becomes an agent-level design axis rather than a serving-layer constraint forced by network latency.
- Per-step guards, novelty checks, and claim grounding become practical once store-plus-embed cost sits near 40 µs, because they no longer need rationing.
- Agent frameworks should expose memory as an allocated loop primitive (create, fill, destroy per task) instead of a long-lived service client configured at startup.
- The residual bottleneck shifts from vector search to embedding placement; local embedders close the last order-of-magnitude gap.
- Restatement-in-context works for tiny working sets but pays growing per-turn token rent; the store's per-fetch pricing takes over as the working set grows.
Where Pith is reading between the lines
- If the dose-response holds, multi-agent systems co-located on one host could share an in-process blackboard as common working memory without network tax, changing how collaborative agents are wired.
- Long-horizon agents whose steps-per-turn grow will see the edge of in-loop guards over per-turn RAG widen, so the paper's next long-horizon study is the natural place the claim becomes or fails to become operationally decisive.
- Read-policy failures (one bounded recall through k=8 while the store holds more) suggest that once latency is free, the next engineering problem is not storage reliability but how the loop decides what to surface.
- The same latency criterion that classifies an in-process store as working memory for an LLM would classify a 100 ms store as working memory for a much slower cognizer, predicting a continuous rather than absolute boundary.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that retrieval latency is a property of store location rather than of the in-loop pattern itself, and that an in-process store (~100 µs) makes per-step read/write affordable enough to treat external memory as extended working memory. It formalizes a feasibility frontier (Eqs. 1–2), reinterprets the extended-mind parity principle as a latency-plus-wiring criterion, and isolates a causal path: under a fixed per-turn memory budget B, only store answer speed is varied, and redundant actions rise monotonically with latency (0.0 of 12 at in-process speed to 7.2 of 12 at +110 ms; exact permutation p=0.0079 on two models). An end-to-end windowed five-fact recall task shows memory tools recovering 3.6–4.8/5 from a 0/5 baseline floor, with store ops at p50 80–165 µs and zero store losses (244/244 writes); an instructed restate-every-reply baseline also scores 5/5. The residual bottleneck is identified as network embedding and closed to a measured ~40 µs complete op with a local embedder.
Significance. If the result holds, it cleanly relocates a design constraint that the agent-memory and RAG literatures have treated as fixed: per-step retrieval frequency becomes an agent-level design axis once the store is in-process, rather than a serving-layer or once-per-turn rationing problem. The causal isolation under a fixed budget (identical store answers across arms; damage only through unaffordable checks), exact permutation tests, zero guard errors where defined, honest reporting that restatement beats tools on the small task, and the measured local-embedder stack are genuine strengths. The work is primarily systems/engineering rather than a new cognitive theory; the parity framing is optional motivation and does not carry the empirical claims. The contribution is real for agent framework design if the affordability result generalizes beyond the two task families shown.
major comments (3)
- [§5.1, Table 4, abstract] §5.1 and Table 4: The primary causal claim is budget-mediated by construction. Latency never changes store answers or model decisions; it only changes how many times the identical correct match is allowed to fire under the hard abort policy (lookups whose estimated cost exceeds remaining B are skipped). The monotone 0.0→7.2 curve is therefore floor(B/c) applied to the residual duplicates the models miss (~7 of 12). The manuscript states the conditionality in places (“under a memory budget below one store round trip…”; “latency does not change what the model knows; it changes what the engineer can afford to wire”), but the abstract and §6.1 framing (“store latency alone changes the outcome”) can still be over-read as latency affecting cognition rather than scaffolding affordability. Tighten the claim language throughout so the result is stated as: under a hard per-turn memory budget, stor
- [§6 Table 1, §2] §6 Table 1 and §2 rent argument: On the five-fact task the instructed restate-every-reply baseline scores 5.0 in all twenty runs and beats memory tools (3.6–4.8/5). The paper correctly reports this and argues restatement pays per-turn rent that grows with the working set, but that scaling claim is not measured—only argued. For the end-to-end demonstration to support “memory in the loop” as the preferred mechanism rather than disciplined restatement, either (i) add a working-set size sweep where restatement token cost or failure is observed, or (ii) explicitly scope the recall result as “memory recovers the designed window floor; restatement also works at this scale and is cheaper to implement until the working set grows.” Leaving the scaling unmeasured while using the small-task win for the store weakens contribution (4).
- [§3, §6.1] §3 “What per-step memory unlocks” lists working-memory offload, novelty checks, loop/action guards, and per-step grounding as newly practical capabilities. Only the loop guard (§5.1/§6.1) and a single bounded recall (§6) are demonstrated. The guard is the simplest member of that family and is well executed; the broader list is currently promissory. Either narrow the unlocks claim to what is measured, or add at least one additional per-step pattern (e.g., novelty or claim grounding) under the same latency manipulation so the regime claim is not carried by a single scaffold pattern.
minor comments (6)
- [§6] §6 memory misses: All misses are attributed to over-writing plus a single k=8 read. k is a fixed design constant, not swept. A short note or appendix cell with k∈{8,16} or repeated recall would show how much of the 3.6–4.8 gap is read-policy rather than substrate, and would strengthen the “store never failed” separation.
- [Table 2, §5.1, §6.1] Table 2 vs Table 3: Table 2 uses op≈85 µs; the deterministic guard text later cites 17.5 µs on regeneration and 85 µs in the audit run. Drift is disclosed in §5.1, but the main tables should pin one host/run-of-record consistently or report both with a clear label so readers do not reconcile conflicting floors.
- [Figure 2, §3, §7] Figure 2 feasibility frontier: State explicitly whether c includes only store+embed or also any serialization/tool-wrapper overhead measured in the live agent path. The ~116 µs vs ~40 µs complete-op numbers in §3 vs §7 need a one-line reconciliation (working-set size / which p50).
- [§4] §4 parity discussion is long relative to its load-bearing role. The manuscript already says the engineering results stand without the philosophy; consider moving the Adams–Aizawa and Library Theorem engagement to an appendix so the causal and latency results surface earlier for systems readers.
- [§5.1] Window units differ across tasks (messages vs exchanges). A single clarifying sentence at first use in §5.1 would prevent miscomparison of w=4 across the trip and guard experiments.
- [References] References include several 2026 arXiv preprints and industry posts. Ensure final citation keys and DOIs are stable at camera-ready; practitioner URLs (Mem0, Redis, Ryjox) should be archived (e.g., perma.cc) if the journal requires durable links.
Circularity Check
No significant circularity: latency measurements, feasibility arithmetic, and the budget-mediated dose-response are self-contained empirical/engineering results, not predictions forced by definition or self-citation.
full rationale
The paper’s load-bearing chain does not reduce to its inputs by construction in the sense this pass targets. (1) Store and embed latencies are measured (in-process p50 80–165 µs; Qdrant loopback; network embed 200–400 ms; local embed+store ~40 µs), not fitted to the outcomes they are later used to explain. (2) The feasibility frontier (Eqs. 1–2, f_max) is transparent cost arithmetic from those measurements under a stated β budget share—not a prediction of task accuracy. (3) The causal loop-guard experiment (§5.1, Table 4) deliberately gates lookups by a fixed per-turn budget B so that injected latency only changes affordability; the paper states this path explicitly (“latency … alters only whether a lookup fits the budget”; “arithmetic made visible” for the deterministic panel) and does not present the floor(B/c) relation as a first-principles derivation of cognition. Independent empirical content remains: real models miss ~7/12 disguised duplicates from the window, the semantic guard has measured precision/recall 1.0 on extracted action lines, intermediate rungs and the per-turn RAG arm (c) are not pure arithmetic, and the trip-planning recall matrix (0/5 → 3.6–4.8/5; notes baseline 5/5) is a separate tool-use experiment. (4) The extended-mind / parity framing (§4) is explicitly non-load-bearing for the numbers (“engineering contribution … does not depend on any philosophical reading”). (5) Author development of vxdb is disclosed as instrument COI, not used as a uniqueness theorem or self-cited premise that forbids alternatives. No self-definitional fit-as-prediction, no uniqueness imported from overlapping authors, no ansatz smuggled via self-citation. Score 0 is the proportionate finding.
Axiom & Free-Parameter Ledger
free parameters (5)
- per-turn memory budget B =
100 ms (operating); grid {25,50,100,200,400} and {100,150,250,500} ms
- latency budget share β =
0.1
- context window w =
4
- recall k =
8
- semantic match threshold =
0.96
axioms (4)
- domain assumption End-to-end latency approximately sums per-step (t_reason + f·(t_embed + t_store))
- ad hoc to paper Clark & Chalmers parity criteria (constancy, direct availability, automatic endorsement) can be read as a latency budget plus loop wiring
- domain assumption A fixed per-turn memory dead-time budget is a legitimate designer constraint under which latency alone can price a guard out of existence
- domain assumption CoALA-style modular memory with reasoning/retrieval/learning actions is an appropriate agent frame
invented entities (2)
-
memory-in-the-loop regime (per-step retrieval frequency)
independent evidence
-
WorkingMemory in-process primitive
independent evidence
read the original abstract
Language agents run a loop - observe, reason, act - but the memory they reason over sits outside it: a store queried at most once per turn. We study the regime where memory moves inside the loop, read and written on every step. The obstacle has always been latency: networked stores answer in tens to hundreds of milliseconds, and in-loop retrieval can inflate end-to-end latency by up to 83x when retrieval is expensive. Prior work manages that cost rather than questioning it: serving-layer scheduling hides it, "memory-first" designs ration retrieval to once per turn. We argue latency is a property of where the store lives, not the in-loop pattern: an in-process store answers in ~100us, three orders of magnitude below the network regime, and at that speed the per-step tax collapses. By the extended-mind thesis's parity principle, a store fast enough to be constantly and directly available becomes extended working memory, not a tool the agent merely consults. The premise is causal: holding a fixed per-turn memory-latency budget and varying only the store's answer speed, redundant actions rise monotonically with latency - 0.0 of 12 at in-process speed, 7.2 of 12 at a 110ms cloud round trip (gpt-5-nano, gpt-5-mini; exact permutation p=0.0079). We demonstrate the regime end-to-end: across four GPT-5-class models under a bounded window, recall improves from 0/5 to 3.6-4.8/5 with in-loop memory, store ops at p50 80-165us - though an instructed restate-every-reply baseline also solves it perfectly, at a token cost that grows with the working set. The store never lost a fact in any run (244 of 244 writes kept); every miss traces to the agent's read policy, not the store. Our measurements also relocate the bottleneck: the dominant per-step cost is embedding (~200-400ms over the network); pairing the in-process store with a small local embedder returns the complete operation to a measured ~40us.
Figures
Reference graph
Works this paper leans on
-
[1]
Emmanuel Bamidele. AMV-L: Lifecycle-managed agent memory for tail-latency control in long- running LLM systems.arXiv preprint arXiv:2603.04443,
-
[2]
Yuxuan Cai, Wei Li, Jie Zhou, Qin Chen, Xin Li, Bo Zhang, and Liang He. Ask only when needed: Proactive retrieval from memory and skills for experience-driven lifelong agents.arXiv preprint arXiv:2604.20572,
work page internal anchor Pith review Pith/arXiv arXiv
-
[3]
Memory for autonomous LLM agents: Mechanisms, evaluation, and emerging frontiers
Pengfei Du. Memory for autonomous LLM agents: Mechanisms, evaluation, and emerging frontiers. arXiv preprint arXiv:2603.07670,
-
[4]
A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
arXiv:2405.06211. Rénald Gesnot. The impact of artificial intelligence on human thought.arXiv preprint arXiv:2508.16628,
work page internal anchor Pith review Pith/arXiv arXiv
-
[5]
Empowering Working Memory for Large Language Model Agents
Jing Guo, Nan Li, Jianchuan Qi, Hang Yang, Ruiqiao Li, Yuzhen Feng, Si Zhang, and Ming Xu. Empowering working memory for large language model agents.arXiv preprint arXiv:2312.17259,
work page internal anchor Pith review Pith/arXiv arXiv
-
[6]
Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
Yuanzhe Hu, Yu Wang, and Julian McAuley. Evaluating memory in LLM agents via incremental multi-turn interactions.arXiv preprint arXiv:2507.05257, 2025a. Yuyang Hu, Shichun Liu, Yanwei Yue, Guibin Zhang, Boyang Liu, et al. Memory in the age of AI agents.arXiv preprint arXiv:2512.13564, 2025b. Hao Kang, Qingru Zhang, Han Cai, Weiyuan Xu, Tushar Krishna, Yil...
work page internal anchor Pith review Pith/arXiv arXiv
-
[7]
Lost in the Middle: How Language Models Use Long Contexts
arXiv:2307.03172. 16 Zachary F. Mainen. The library theorem: How external organization governs agentic reasoning capacity.arXiv preprint arXiv:2603.21272,
work page internal anchor Pith review Pith/arXiv arXiv
-
[8]
Jakob Nielsen.Usability Engineering
DOI 10.1145/1476589.1476628. Jakob Nielsen.Usability Engineering. Academic Press,
-
[9]
Agent Memory: Characterization and System Implications of Stateful Long-Horizon Workloads
Yasmine Omri, Ziyu Gan, Zachary Broveak, Robin Geens, Zexue He, Alex Pentland, Marian Verhelst, Tsachy Weissman, and Thierry Tambe. Agent memory: Characterization and system implications of stateful long-horizon workloads.arXiv preprint arXiv:2606.06448,
work page internal anchor Pith review Pith/arXiv arXiv
-
[10]
MemGPT: Towards LLMs as Operating Systems
Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. MemGPT: Towards LLMs as operating systems.arXiv preprint arXiv:2310.08560,
work page internal anchor Pith review Pith/arXiv arXiv
-
[11]
Generative Agents: Interactive Simulacra of Human Behavior
arXiv:2304.03442. Elija Perrier. Operationalising extended cognition: Formal metrics for corporate knowledge and legal accountability.arXiv preprint arXiv:2510.16193,
work page internal anchor Pith review Pith/arXiv arXiv
-
[12]
Invisible Architectures of Thought: Toward a New Science of AI as Cognitive Infrastructure
Giuseppe Riva. Invisible architectures of thought: Toward a new science of AI as cognitive infras- tructure.arXiv preprint arXiv:2507.22893,
work page internal anchor Pith review Pith/arXiv arXiv
-
[13]
Reflexion: Language Agents with Verbal Reinforcement Learning
arXiv:2303.11366. Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. Cognitive architec- tures for language agents.Transactions on Machine Learning Research,
work page internal anchor Pith review Pith/arXiv arXiv
-
[14]
Cognitive Architectures for Language Agents
arXiv:2309.02427. Haoran Tan, Zeyu Zhang, Chen Ma, Xu Chen, Quanyu Dai, and Zhenhua Dong. MemBench: Towards more comprehensive evaluation on the memory of LLM-based agents.arXiv preprint arXiv:2506.21605,
work page internal anchor Pith review Pith/arXiv arXiv
-
[15]
Voyager: An Open-Ended Embodied Agent with Large Language Models
arXiv:2305.16291. Yaxiong Wu, Sheng Liang, Chen Zhang, Yichao Wang, Yongyue Zhang, Huifeng Guo, Ruiming Tang, and Yong Liu. From human memory to AI memory: A survey on memory mechanisms in the era of LLMs.arXiv preprint arXiv:2504.15965,
work page internal anchor Pith review Pith/arXiv arXiv
-
[16]
Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agents
Tiannuo Yang, Zebin Yao, Bowen Jin, Lixiao Cui, Yusen Li, Gang Wang, and Xiaoguang Liu. Demystifying and enhancing the efficiency of large language model based search agents.arXiv preprint arXiv:2505.12065,
work page internal anchor Pith review Pith/arXiv arXiv
-
[17]
ReAct: Synergizing Reasoning and Acting in Language Models
arXiv:2210.03629. 17 Aojie Yuan, Haiyue Zhang, and Shahin Nazarian. AgentIR: A workload-adaptive cascade retrieval substrate for long-term conversational memory.arXiv preprint arXiv:2605.25092,
work page internal anchor Pith review Pith/arXiv arXiv
-
[18]
Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
Chenyu Zhou, Huacan Chai, Wenteng Chen, Zihan Guo, Rong Shan, Yuanyi Song, Tianyi Xu, Yingx- uan Yang, Aofan Yu, Weiming Zhang, Congming Zheng, Jiachen Zhu, Zeyu Zheng, Zhuosheng Zhang, Xingyu Lou, Changwang Zhang, Zhihui Fu, Jun Wang, Weiwen Liu, Jianghao Lin, and Weinan Zhang. Externalization in LLM agents: A unified review of memory, skills, protocols ...
work page internal anchor Pith review Pith/arXiv arXiv
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.