Pith. sign in

REVIEW 3 major objections 6 minor 18 references

Store latency alone decides whether an agent can use memory on every reasoning step, and that changes what the agent actually does.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · grok-4.5

2026-07-11 03:43 UTC pith:WMPDCHI5

load-bearing objection Clean causal isolation of store latency under a fixed budget, with honest baselines and real microsecond measurements; the philosophy is optional packaging. the 3 major comments →

arxiv 2607.05690 v1 pith:WMPDCHI5 submitted 2026-07-06 cs.AI cs.CL

Memory in the Loop: In-Process Retrieval as ExtendedWorking Memory for Language Agents

classification cs.AI cs.CL
keywords language agentsworking memoryin-process retrievalretrieval frequencystore latencyextended mindloop guardembedding bottleneck
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Language agents reason in a loop, but their memory usually sits outside it: fetched once per turn from a networked store that answers in tens to hundreds of milliseconds. This paper argues that the cost of in-loop memory is not a law of the pattern itself but a property of where the store lives. An in-process store answers in about 100 microseconds, three orders of magnitude faster, so the per-step tax collapses and memory can be read and written on every step. The authors ground the claim in the extended-mind parity principle: a store that is constantly and directly available becomes eligible as extended working memory rather than a tool the agent consults. They then show the premise is causal. Under a fixed per-turn memory budget, varying only the store's answer speed makes redundant actions rise from zero at in-process speed to 7.2 of 12 at a 110 ms cloud round trip. End-to-end, in-loop memory lifts recall on a bounded-window task from 0/5 to 3.6–4.8/5, store operations stay in the 80–165 microsecond range, and the remaining bottleneck—network embedding—falls to about 40 microseconds with a small local embedder. A sympathetic reader cares because this reframes retrieval frequency as a design variable rather than a constraint, and opens a class of per-step checks (guards, novelty, grounding) that network latency had made unaffordable.

Core claim

Holding a fixed per-turn memory-latency budget and varying only the store's answer speed, redundant actions rise monotonically with latency: 0.0 of 12 at in-process speed versus 7.2 of 12 at a 110 ms cloud round trip. Store latency alone therefore determines whether a per-step loop guard can exist and thereby changes task outcomes. At in-process speeds (~100 µs), memory moves inside the agent loop and becomes eligible as extended working memory; network stores force rationing to once per turn.

What carries the argument

Memory in the loop: per-step read/write of an in-process associative store, gated by a feasibility frontier that prices affordable retrieval frequency as f_max = (β/(1-β)) · (r/c). The causal instrument is a fixed per-turn memory budget B under which only store answer speed is varied, so latency changes whether a lookup fits rather than what it returns.

Load-bearing premise

That a designer-chosen fixed memory budget per turn (about 100 ms, or 10% of a one-second step) is the right way to expose the causal path, and that a semantic loop-guard task under a short window stands in for the broader class of per-step memory uses the paper claims become practical once latency collapses.

What would settle it

Rerun the real-LLM loop-guard experiment with identical models, prompts, and seeded workloads, but replace the injected-delay ladder with a live cloud vector store at measured ~110 ms RTT under the same 100 ms per-turn budget: if redundant investigations no longer rise to ~7.2 of 12 and the in-process arm no longer sits at zero, the causal claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Retrieval frequency (per-turn vs per-step) becomes an agent-level design axis rather than a serving-layer constraint forced by network latency.
  • Per-step guards, novelty checks, and claim grounding become practical once store-plus-embed cost sits near 40 µs, because they no longer need rationing.
  • Agent frameworks should expose memory as an allocated loop primitive (create, fill, destroy per task) instead of a long-lived service client configured at startup.
  • The residual bottleneck shifts from vector search to embedding placement; local embedders close the last order-of-magnitude gap.
  • Restatement-in-context works for tiny working sets but pays growing per-turn token rent; the store's per-fetch pricing takes over as the working set grows.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the dose-response holds, multi-agent systems co-located on one host could share an in-process blackboard as common working memory without network tax, changing how collaborative agents are wired.
  • Long-horizon agents whose steps-per-turn grow will see the edge of in-loop guards over per-turn RAG widen, so the paper's next long-horizon study is the natural place the claim becomes or fails to become operationally decisive.
  • Read-policy failures (one bounded recall through k=8 while the store holds more) suggest that once latency is free, the next engineering problem is not storage reliability but how the loop decides what to surface.
  • The same latency criterion that classifies an in-process store as working memory for an LLM would classify a 100 ms store as working memory for a much slower cognizer, predicting a continuous rather than absolute boundary.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that retrieval latency is a property of store location rather than of the in-loop pattern itself, and that an in-process store (~100 µs) makes per-step read/write affordable enough to treat external memory as extended working memory. It formalizes a feasibility frontier (Eqs. 1–2), reinterprets the extended-mind parity principle as a latency-plus-wiring criterion, and isolates a causal path: under a fixed per-turn memory budget B, only store answer speed is varied, and redundant actions rise monotonically with latency (0.0 of 12 at in-process speed to 7.2 of 12 at +110 ms; exact permutation p=0.0079 on two models). An end-to-end windowed five-fact recall task shows memory tools recovering 3.6–4.8/5 from a 0/5 baseline floor, with store ops at p50 80–165 µs and zero store losses (244/244 writes); an instructed restate-every-reply baseline also scores 5/5. The residual bottleneck is identified as network embedding and closed to a measured ~40 µs complete op with a local embedder.

Significance. If the result holds, it cleanly relocates a design constraint that the agent-memory and RAG literatures have treated as fixed: per-step retrieval frequency becomes an agent-level design axis once the store is in-process, rather than a serving-layer or once-per-turn rationing problem. The causal isolation under a fixed budget (identical store answers across arms; damage only through unaffordable checks), exact permutation tests, zero guard errors where defined, honest reporting that restatement beats tools on the small task, and the measured local-embedder stack are genuine strengths. The work is primarily systems/engineering rather than a new cognitive theory; the parity framing is optional motivation and does not carry the empirical claims. The contribution is real for agent framework design if the affordability result generalizes beyond the two task families shown.

major comments (3)
  1. [§5.1, Table 4, abstract] §5.1 and Table 4: The primary causal claim is budget-mediated by construction. Latency never changes store answers or model decisions; it only changes how many times the identical correct match is allowed to fire under the hard abort policy (lookups whose estimated cost exceeds remaining B are skipped). The monotone 0.0→7.2 curve is therefore floor(B/c) applied to the residual duplicates the models miss (~7 of 12). The manuscript states the conditionality in places (“under a memory budget below one store round trip…”; “latency does not change what the model knows; it changes what the engineer can afford to wire”), but the abstract and §6.1 framing (“store latency alone changes the outcome”) can still be over-read as latency affecting cognition rather than scaffolding affordability. Tighten the claim language throughout so the result is stated as: under a hard per-turn memory budget, stor
  2. [§6 Table 1, §2] §6 Table 1 and §2 rent argument: On the five-fact task the instructed restate-every-reply baseline scores 5.0 in all twenty runs and beats memory tools (3.6–4.8/5). The paper correctly reports this and argues restatement pays per-turn rent that grows with the working set, but that scaling claim is not measured—only argued. For the end-to-end demonstration to support “memory in the loop” as the preferred mechanism rather than disciplined restatement, either (i) add a working-set size sweep where restatement token cost or failure is observed, or (ii) explicitly scope the recall result as “memory recovers the designed window floor; restatement also works at this scale and is cheaper to implement until the working set grows.” Leaving the scaling unmeasured while using the small-task win for the store weakens contribution (4).
  3. [§3, §6.1] §3 “What per-step memory unlocks” lists working-memory offload, novelty checks, loop/action guards, and per-step grounding as newly practical capabilities. Only the loop guard (§5.1/§6.1) and a single bounded recall (§6) are demonstrated. The guard is the simplest member of that family and is well executed; the broader list is currently promissory. Either narrow the unlocks claim to what is measured, or add at least one additional per-step pattern (e.g., novelty or claim grounding) under the same latency manipulation so the regime claim is not carried by a single scaffold pattern.
minor comments (6)
  1. [§6] §6 memory misses: All misses are attributed to over-writing plus a single k=8 read. k is a fixed design constant, not swept. A short note or appendix cell with k∈{8,16} or repeated recall would show how much of the 3.6–4.8 gap is read-policy rather than substrate, and would strengthen the “store never failed” separation.
  2. [Table 2, §5.1, §6.1] Table 2 vs Table 3: Table 2 uses op≈85 µs; the deterministic guard text later cites 17.5 µs on regeneration and 85 µs in the audit run. Drift is disclosed in §5.1, but the main tables should pin one host/run-of-record consistently or report both with a clear label so readers do not reconcile conflicting floors.
  3. [Figure 2, §3, §7] Figure 2 feasibility frontier: State explicitly whether c includes only store+embed or also any serialization/tool-wrapper overhead measured in the live agent path. The ~116 µs vs ~40 µs complete-op numbers in §3 vs §7 need a one-line reconciliation (working-set size / which p50).
  4. [§4] §4 parity discussion is long relative to its load-bearing role. The manuscript already says the engineering results stand without the philosophy; consider moving the Adams–Aizawa and Library Theorem engagement to an appendix so the causal and latency results surface earlier for systems readers.
  5. [§5.1] Window units differ across tasks (messages vs exchanges). A single clarifying sentence at first use in §5.1 would prevent miscomparison of w=4 across the trip and guard experiments.
  6. [References] References include several 2026 arXiv preprints and industry posts. Ensure final citation keys and DOIs are stable at camera-ready; practitioner URLs (Mem0, Redis, Ryjox) should be archived (e.g., perma.cc) if the journal requires durable links.

Circularity Check

0 steps flagged

No significant circularity: latency measurements, feasibility arithmetic, and the budget-mediated dose-response are self-contained empirical/engineering results, not predictions forced by definition or self-citation.

full rationale

The paper’s load-bearing chain does not reduce to its inputs by construction in the sense this pass targets. (1) Store and embed latencies are measured (in-process p50 80–165 µs; Qdrant loopback; network embed 200–400 ms; local embed+store ~40 µs), not fitted to the outcomes they are later used to explain. (2) The feasibility frontier (Eqs. 1–2, f_max) is transparent cost arithmetic from those measurements under a stated β budget share—not a prediction of task accuracy. (3) The causal loop-guard experiment (§5.1, Table 4) deliberately gates lookups by a fixed per-turn budget B so that injected latency only changes affordability; the paper states this path explicitly (“latency … alters only whether a lookup fits the budget”; “arithmetic made visible” for the deterministic panel) and does not present the floor(B/c) relation as a first-principles derivation of cognition. Independent empirical content remains: real models miss ~7/12 disguised duplicates from the window, the semantic guard has measured precision/recall 1.0 on extracted action lines, intermediate rungs and the per-turn RAG arm (c) are not pure arithmetic, and the trip-planning recall matrix (0/5 → 3.6–4.8/5; notes baseline 5/5) is a separate tool-use experiment. (4) The extended-mind / parity framing (§4) is explicitly non-load-bearing for the numbers (“engineering contribution … does not depend on any philosophical reading”). (5) Author development of vxdb is disclosed as instrument COI, not used as a uniqueness theorem or self-cited premise that forbids alternatives. No self-definitional fit-as-prediction, no uniqueness imported from overlapping authors, no ansatz smuggled via self-citation. Score 0 is the proportionate finding.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 2 invented entities

The central causal claim rests on a small set of design choices (budget B, budget share β, window sizes, match threshold, k) that are stated and partly swept, plus standard background from cognitive science and agent architectures. No new physical entities are postulated; “memory in the loop” and WorkingMemory are named regimes/primitives. The largest non-empirical load is the mapping of Clark & Chalmers’ parity criteria onto a latency budget and loop wiring, which the paper treats as an engineering criterion rather than a theorem.

free parameters (5)
  • per-turn memory budget B = 100 ms (operating); grid {25,50,100,200,400} and {100,150,250,500} ms
    Operating point B=100 ms (and sweeps 25–500 ms) chosen to match β=0.1 of a 1 s interactive step and classic human interface thresholds; the cliff location depends on B.
  • latency budget share β = 0.1
    Fraction of end-to-end time allowed for memory; used to define f_max. Ordering is β-invariant but absolute affordability is not.
  • context window w = 4
    Trip task w=4 messages; guard task w=4 exchanges; chosen inside the failure region by design (window×facts sweep).
  • recall k = 8
    Fixed k=8 for the single per-run recall; outgrowing k is necessary for observed misses.
  • semantic match threshold = 0.96
    Calibrated offline on held-out seeds to F1=1.0 at 0.96 for action-line matching; design choice that makes the guard effectively exact on extracted lines.
axioms (4)
  • domain assumption End-to-end latency approximately sums per-step (t_reason + f·(t_embed + t_store))
    Cost model Eq. (1) in §3; standard additive blocking model for sequential agent steps.
  • ad hoc to paper Clark & Chalmers parity criteria (constancy, direct availability, automatic endorsement) can be read as a latency budget plus loop wiring
    §4 reinterprets the first two criteria as latency and the third as scaffolding; engineering claim does not depend on the philosophy but the constitutive-working-memory framing does.
  • domain assumption A fixed per-turn memory dead-time budget is a legitimate designer constraint under which latency alone can price a guard out of existence
    §5.1; without some budget the causal path from latency to unaffordable checks disappears.
  • domain assumption CoALA-style modular memory with reasoning/retrieval/learning actions is an appropriate agent frame
    §2 background; frequency of retrieval is left open by CoALA and is the variable the paper adds.
invented entities (2)
  • memory-in-the-loop regime (per-step retrieval frequency) independent evidence
    purpose: Name the design axis of reading/writing the store on every reasoning step rather than once per turn
    Definition in §3; not a physical entity but a named regime whose feasibility the paper measures.
  • WorkingMemory in-process primitive independent evidence
    purpose: Ephemeral allocate-not-connect semantic store instrumented for per-op latency
    §5 method; implemented on vxdb (author-developed). Independent evidence is the latency measurements themselves.

pith-pipeline@v1.1.0-grok45 · 22565 in / 3723 out tokens · 34817 ms · 2026-07-11T03:43:06.237128+00:00 · methodology

0 comments
read the original abstract

Language agents run a loop - observe, reason, act - but the memory they reason over sits outside it: a store queried at most once per turn. We study the regime where memory moves inside the loop, read and written on every step. The obstacle has always been latency: networked stores answer in tens to hundreds of milliseconds, and in-loop retrieval can inflate end-to-end latency by up to 83x when retrieval is expensive. Prior work manages that cost rather than questioning it: serving-layer scheduling hides it, "memory-first" designs ration retrieval to once per turn. We argue latency is a property of where the store lives, not the in-loop pattern: an in-process store answers in ~100us, three orders of magnitude below the network regime, and at that speed the per-step tax collapses. By the extended-mind thesis's parity principle, a store fast enough to be constantly and directly available becomes extended working memory, not a tool the agent merely consults. The premise is causal: holding a fixed per-turn memory-latency budget and varying only the store's answer speed, redundant actions rise monotonically with latency - 0.0 of 12 at in-process speed, 7.2 of 12 at a 110ms cloud round trip (gpt-5-nano, gpt-5-mini; exact permutation p=0.0079). We demonstrate the regime end-to-end: across four GPT-5-class models under a bounded window, recall improves from 0/5 to 3.6-4.8/5 with in-loop memory, store ops at p50 80-165us - though an instructed restate-every-reply baseline also solves it perfectly, at a token cost that grows with the working set. The store never lost a fact in any run (244 of 244 writes kept); every miss traces to the agent's read policy, not the store. Our measurements also relocate the bottleneck: the dominant per-step cost is embedding (~200-400ms over the network); pairing the in-process store with a small local embedder returns the complete operation to a measured ~40us.

Figures

Figures reproduced from arXiv: 2607.05690 by Carlo Lipizzi, Yusuf Khan.

Figure 1
Figure 1. Figure 1: Retrieval frequency. Left: per-turn retrieval (a single network fetch, then an inner loop with no memory access). Right: memory in the loop—an in-process store read and written on every step. retrieval goes to the fact by meaning. The cliff : a long-lived agent eventually exceeds any window, and the failure is abrupt—our window×facts sweep (§10) locates task failure exactly where the fact span exceeds the … view at source ↗
Figure 2
Figure 2. Figure 2: The feasibility frontier: affordable in-loop retrievals per step ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Latency as the cognitive boundary. An in-process store ( [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: In-loop retrieval tax vs. store round-trip latency (log [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Mechanism demonstration under a fixed memory budget. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Dose-response: redundant investigations vs. injected per-lookup store latency under a fixed [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The window cliff (sensitivity analysis): task pass rate across facts-required [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages · 15 internal anchors

  1. [1]

    AMV-L: Lifecycle-managed agent memory for tail-latency control in long- running LLM systems.arXiv preprint arXiv:2603.04443,

    Emmanuel Bamidele. AMV-L: Lifecycle-managed agent memory for tail-latency control in long- running LLM systems.arXiv preprint arXiv:2603.04443,

  2. [2]

    Ask Only When Needed: Proactive Retrieval from Memory and Skills for Experience-Driven Lifelong Agents

    Yuxuan Cai, Wei Li, Jie Zhou, Qin Chen, Xin Li, Bo Zhang, and Liang He. Ask only when needed: Proactive retrieval from memory and skills for experience-driven lifelong agents.arXiv preprint arXiv:2604.20572,

  3. [3]

    Memory for autonomous LLM agents: Mechanisms, evaluation, and emerging frontiers

    Pengfei Du. Memory for autonomous LLM agents: Mechanisms, evaluation, and emerging frontiers. arXiv preprint arXiv:2603.07670,

  4. [4]

    A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models

    arXiv:2405.06211. Rénald Gesnot. The impact of artificial intelligence on human thought.arXiv preprint arXiv:2508.16628,

  5. [5]

    Empowering Working Memory for Large Language Model Agents

    Jing Guo, Nan Li, Jianchuan Qi, Hang Yang, Ruiqiao Li, Yuzhen Feng, Si Zhang, and Ming Xu. Empowering working memory for large language model agents.arXiv preprint arXiv:2312.17259,

  6. [6]

    Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions

    Yuanzhe Hu, Yu Wang, and Julian McAuley. Evaluating memory in LLM agents via incremental multi-turn interactions.arXiv preprint arXiv:2507.05257, 2025a. Yuyang Hu, Shichun Liu, Yanwei Yue, Guibin Zhang, Boyang Liu, et al. Memory in the age of AI agents.arXiv preprint arXiv:2512.13564, 2025b. Hao Kang, Qingru Zhang, Han Cai, Weiyuan Xu, Tushar Krishna, Yil...

  7. [7]

    Lost in the Middle: How Language Models Use Long Contexts

    arXiv:2307.03172. 16 Zachary F. Mainen. The library theorem: How external organization governs agentic reasoning capacity.arXiv preprint arXiv:2603.21272,

  8. [8]

    Jakob Nielsen.Usability Engineering

    DOI 10.1145/1476589.1476628. Jakob Nielsen.Usability Engineering. Academic Press,

  9. [9]

    Agent Memory: Characterization and System Implications of Stateful Long-Horizon Workloads

    Yasmine Omri, Ziyu Gan, Zachary Broveak, Robin Geens, Zexue He, Alex Pentland, Marian Verhelst, Tsachy Weissman, and Thierry Tambe. Agent memory: Characterization and system implications of stateful long-horizon workloads.arXiv preprint arXiv:2606.06448,

  10. [10]

    MemGPT: Towards LLMs as Operating Systems

    Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. MemGPT: Towards LLMs as operating systems.arXiv preprint arXiv:2310.08560,

  11. [11]

    Generative Agents: Interactive Simulacra of Human Behavior

    arXiv:2304.03442. Elija Perrier. Operationalising extended cognition: Formal metrics for corporate knowledge and legal accountability.arXiv preprint arXiv:2510.16193,

  12. [12]

    Invisible Architectures of Thought: Toward a New Science of AI as Cognitive Infrastructure

    Giuseppe Riva. Invisible architectures of thought: Toward a new science of AI as cognitive infras- tructure.arXiv preprint arXiv:2507.22893,

  13. [13]

    Reflexion: Language Agents with Verbal Reinforcement Learning

    arXiv:2303.11366. Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. Cognitive architec- tures for language agents.Transactions on Machine Learning Research,

  14. [14]

    Cognitive Architectures for Language Agents

    arXiv:2309.02427. Haoran Tan, Zeyu Zhang, Chen Ma, Xu Chen, Quanyu Dai, and Zhenhua Dong. MemBench: Towards more comprehensive evaluation on the memory of LLM-based agents.arXiv preprint arXiv:2506.21605,

  15. [15]

    Voyager: An Open-Ended Embodied Agent with Large Language Models

    arXiv:2305.16291. Yaxiong Wu, Sheng Liang, Chen Zhang, Yichao Wang, Yongyue Zhang, Huifeng Guo, Ruiming Tang, and Yong Liu. From human memory to AI memory: A survey on memory mechanisms in the era of LLMs.arXiv preprint arXiv:2504.15965,

  16. [16]

    Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agents

    Tiannuo Yang, Zebin Yao, Bowen Jin, Lixiao Cui, Yusen Li, Gang Wang, and Xiaoguang Liu. Demystifying and enhancing the efficiency of large language model based search agents.arXiv preprint arXiv:2505.12065,

  17. [17]

    ReAct: Synergizing Reasoning and Acting in Language Models

    arXiv:2210.03629. 17 Aojie Yuan, Haiyue Zhang, and Shahin Nazarian. AgentIR: A workload-adaptive cascade retrieval substrate for long-term conversational memory.arXiv preprint arXiv:2605.25092,

  18. [18]

    Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering

    Chenyu Zhou, Huacan Chai, Wenteng Chen, Zihan Guo, Rong Shan, Yuanyi Song, Tianyi Xu, Yingx- uan Yang, Aofan Yu, Weiming Zhang, Congming Zheng, Jiachen Zhu, Zeyu Zheng, Zhuosheng Zhang, Xingyu Lou, Changwang Zhang, Zhihui Fu, Jun Wang, Weiwen Liu, Jianghao Lin, and Weinan Zhang. Externalization in LLM agents: A unified review of memory, skills, protocols ...