SwiftMem: Fast Agentic Memory via Query-aware Indexing

Anxin Tian; Hui-Ling Zhen; Lei Chen; Mingxuan Yuan; Xianzhi Yu; Xing Li; Yiming Li; Zhenhua Dong

REVIEW 6 major objections 5 minor 2 cited by

The paper claims that agentic memory search can be made query-aware — indexing time and topic so each query touches only the relevant slice of history — cutting reported latency to roughly 11 ms from about 800 ms while holding accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 10:51 UTC pith:XB7HOUHH

load-bearing objection Plausible query-aware memory system with real latency wins, but the current evidence is under-built: no code, borrowed baselines, no tag-recall measurement, and a sloppy PDF. the 6 major comments →

arxiv 2601.08160 v2 pith:XB7HOUHH submitted 2026-01-13 cs.CL cs.AI

SwiftMem: Fast Agentic Memory via Query-aware Indexing

Anxin Tian , Yiming Li , Xing Li , Hui-Ling Zhen , Lei Chen , Xianzhi Yu , Zhenhua Dong , Mingxuan Yuan This is my paper

classification cs.CL cs.AI

keywords agentic memoryquery-aware indexingDAG-tag indextemporal indexretrieval latencyLLM agentsmemory consolidationlong-conversation benchmarks

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Existing agentic memory systems retrieve by scanning every stored episode for each query, so latency grows linearly with conversational history. SwiftMem argues that memory queries have temporal and semantic locality: answers live in specific time ranges and specific topics. It therefore routes each query through two indexes — a binary-searchable timeline and a hierarchy of LLM-generated topic tags — to fetch only the relevant subset before any similarity search. On standard long-conversation benchmarks, the paper reports roughly 10.8–12.7 ms search latency, about 47× faster than strong baselines, with LLM-judge accuracy that remains competitive. The broader claim is that sub-linear retrieval can replace exhaustive retrieval in agent memory without losing answer quality.

Core claim

The central discovery is that the O(N) retrieval bottleneck in agentic memory is avoidable. The paper builds a three-tier index — per-user sorted timelines for logarithmic temporal range queries, a DAG of semantic tags (LLM-generated, hierarchy-aware) that maps a query to k tags and expands them to depth D_max, and an embedding index whose physical layout is periodically reorganized by semantic cluster. Retrieval then only scores episodes in the narrowed subset, giving claimed complexity O(k(log|V|+D_max)) for semantic queries and O(log N_mem) for temporal ones. Empirically, SwiftMem reports sub-15 ms search latency across scales, a 47× speedup over baselines, competitive LLM-judge scores, a

What carries the argument

The load-bearing object is the DAG-Tag index: a directed acyclic graph whose nodes are semantic tags (a hierarchy of conversation topics such as 'pets' with children 'dogs' and 'emotional_support'), where each node stores the episodes it covers, its parent and child relations, and an embedding. A query-tag router selects the top-k tags by cosine similarity to the query embedding, then expands those tags breadth-first up to D_max depth to widen coverage. The temporal index is a per-user sorted list of (timestamp, episode) pairs searched by binary search, with a global episode lookup table for O(1) metadata access. The co-consolidation mechanism periodically rewrites vector storage so episodes

Load-bearing premise

The speedup rests on the assumption that the LLM-generated tags and the query-tag router together never miss the episode a query needs; if a relevant episode is untagged or wrongly routed, it is never searched and recall drops silently.

What would settle it

Compare SwiftMem's retrieved candidate set against exhaustive top-k embedding retrieval on queries whose answers come from episodes that are semantically distant from the query's wording; if evidence recall of the narrowed set falls materially below exhaustive recall while latency stays low, the narrowing is discarding needed memory. A concrete run: inject episodes that the tag generator fails to tag (e.g., rare entities not in the prompt's four semantic dimensions), ask a query whose answer is in one of them, and check whether recall drops.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If the latency numbers hold, long-horizon agents can stay interactive as memory grows, since search cost stops scaling linearly with history length.
Temporal questions ('what did we discuss last week?') become binary-search lookups instead of full scans, which is where much of the reported speedup comes from.
Semantic questions route through the tag DAG, so adding or updating a conversation only requires updating its tags and timeline entries, not re-indexing the whole store.
The co-consolidation result — same evidence recall but higher LLM-judge score after reordering — suggests that physical memory layout affects generation quality, not just speed.
The reported 47× search speedup on long-conversation benchmarks implies the approach could make memory-augmented agents practical in real-time settings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the real test is tag-side recall — whether the LLM tagger plus router ever omits the one episode that holds the answer; measuring that omission rate on a held-out query set would quantify the accuracy risk hidden behind the latency gains.
Editorial inference: the depth parameter D_max and tag count k control the accuracy-latency tradeoff; a sensitivity sweep would show how much recall is bought per extra tag or expansion step.
Editorial inference: the same query-aware narrowing principle could be applied to other memory types the paper mentions (procedural, resource, knowledge vault), or to graph-based memories, wherever a query can be routed to a subset before scoring.
Editorial inference: if consolidation's accuracy gain generalizes, it suggests that ordering evidence by semantic cluster before feeding a generator improves answer coherence — an effect worth testing in plain retrieval pipelines outside agent memory.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Plausible query-aware memory system with real latency wins, but the current evidence is under-built: no code, borrowed baselines, no tag-recall measurement, and a sloppy PDF.

read the letter

SwiftMem is a sensible engineering combination: temporal sorted timelines, a DAG-tag router, and embedding-tag co-consolidation. The individual pieces have precedents, but the specific query-routed architecture is new enough to be worth a look. The ablations on LongMemEval_S are a genuinely good idea — varying the temporal hint ratio and showing latency drop while evidence recall improves is the right kind of experiment. The co-consolidation ablation also shows stable recall with a latency win. If the latency numbers hold, this removes a real bottleneck for deployed agents.

The soft spots are real and they are in the load-bearing parts. The entire speedup rests on the assumption that a small set of DAG tags plus expansion always covers the relevant episodes. The paper never measures tag recall, routing accuracy, or sensitivity to k and D_max. If the router picks the wrong tags, the gold episode is never scored — no retriever can save you. That omission is not minor. Theorem 3.3 also only bounds tag selection and expansion, not the cost of scanning episodes attached to popular tags. A tag like 'pets' could plausibly cover a large fraction of memory, and the paper gives no distribution of episodes per tag. So the sub-linear claim is conditional, and the condition is unverified.

The empirical setup adds risk. Baselines marked with a dagger are imported from Nemori rather than re-run in SwiftMem's environment. Latency comparisons across papers are notoriously sensitive to hardware and index configuration. There are no error bars or repeated runs, and the scalability numbers (12.61 -> 11.55 -> 10.62 ms) look exactly like noise. On accuracy, calling the multi-hop score of 0.589 versus Nemori's 0.762 'competitive' stretches the word; that is a 17-point drop on one of the four LoCoMo categories. The abstract in the submission metadata says 10.8/12.7 ms while the paper's own abstract says '47x faster' — reconcilable, but sloppy. And the PDF contains literal corrupted glyph sequences in the ablation figures, which suggests the camera-ready processing failed somewhere.

None of this means the central idea is wrong. The design is coherent, and query-aware narrowing is a plausible path to sub-linear retrieval in agent memory. But as submitted, the paper asks the reader to trust a lot: borrowed baselines, no code, no tag-recall numbers, and no error bars. It deserves a serious referee, but the referee should require the authors to release code, re-run baselines in the same environment, and report tag-recall and routing accuracy. Without those, the 'orders of magnitude speedup' claim is not verifiable.

I would send this to peer review with a clear request for those additions. If the code and measurements come through, it could be a useful systems contribution. If they don't, the speedup claim should not be accepted on faith.

Referee Report

6 major / 5 minor

Summary. The manuscript proposes SwiftMem, a query-aware agentic memory system that combines a user-specific temporal index, an LLM-generated DAG-based semantic tag index, and an embedding-tag co-consolidation mechanism to narrow memory retrieval to query-relevant subsets. The paper claims sub-linear retrieval complexity, reports a 47× search-latency improvement over HNSW-backed baselines on LoCoMo while maintaining competitive LLM-judge accuracy (e.g., 0.704 vs. 0.792 for Nemori), and includes ablations on LongMemEval_S. The central hypothesis is that query-aware indexing over temporal and semantic locality can remove the full-scope retrieval bottleneck of current agentic memory systems.

Significance. If substantiated, this is a useful and timely direction: memory retrieval latency is a real bottleneck for long-horizon LLM agents, and the proposed design is conceptually simple, builds on standard indexing ideas, and targets both temporal and semantic locality. The paper's formal complexity characterization of tag selection/expansion and its explicit separation of temporal vs. semantic query paths are strengths, as is the use of two established benchmarks and an LLM-judge protocol. However, the current empirical support is not strong enough to establish the headline claims: most baseline numbers are imported from a single prior paper, no tag-coverage or routing-accuracy measurement is reported, and the LongMemEval_S results are limited to ablations.

major comments (6)

[§4.1.3, Tables 1–2] The majority of baseline numbers are not measured in this paper. FullContext, LangMem, Mem0, RAG-4096, and Zep are marked † and 'evaluated using results from Nemori (Nan et al., 2025)' (Table 1 caption). Only Nemori and SwiftMem are run in the present environment. The headline 47× speedup (Table 2: 11 ms vs. 522/835 ms for Zep/Nemori) therefore compares across different hardware, software stacks, and possibly different retrieval configurations. The central efficiency claim requires same-environment re-runs of all baselines, or at minimum a clearly stated hardware-equivalence argument and variance reporting.
[§4.3, Figure 4] The scalability claim rests on three dataset configurations with reported latencies 12.61, 11.55, and 10.62 ms. The non-monotonic trend is within plausible run-to-run noise, yet the text states SwiftMem 'scales sub-linearly' and attributes the full-set latency being lower than subsets to better consolidation. No repeated trials, confidence intervals, or distribution over queries are reported. The experiment should include multiple runs and more granular memory sizes before claiming a scaling law.
[§3.2.2–§3.2.3, §4.4] No tag-recall or routing-accuracy measurement is reported. SwiftMem's speed and accuracy both depend on the LLM-generated DAG tags and the top-k query-tag router covering every episode relevant to a query; if a relevant episode's tags are not selected, it is never scored regardless of the embedding index. The ablations report evidence recall only for the temporal-index setup and for co-consolidation (90.5%), not for tag coverage. The authors should measure tag coverage/recall and run sensitivity analyses for k and Dmax.
[Theorem 3.3, Eq. (5)] The stated query complexity O(k·(log|V|+Dmax)) covers tag selection and DAG expansion, not the cost of scanning or scoring the episodes attached to the selected tags. If a popular tag (e.g., 'pets' in Figure 3) is associated with a large fraction of Nmem, the actual retrieval step can still be O(Nmem). The paper gives no distribution of episode counts per tag and no end-to-end complexity model. This must be addressed for the sub-linear claim to hold.
[Abstract, §4] The abstract claims results 'across LoCoMo and LongMemEval_S' with 10.8/12.7 ms latency, but all comparative tables are on LoCoMo only; LongMemEval_S appears only in the ablation (§4.4), where latencies are 7.2–11.1 ms and no overall benchmark score is reported. The abstract's numerical claims are not traceable to a table or figure. Full LongMemEval_S results should be added, and all reported numbers should be reconciled with the tables.
[§4.4] The co-consolidation ablation shows the LLM Judge score rising from 64.3% to 78.6% while evidence recall stays at 90.5%. Without additional analysis, this 14-point gain in judge score without a corresponding recall gain is unexplained; it could reflect judge sensitivity to context ordering rather than retrieval quality. The paper should investigate this or temper the claim that consolidation improves semantic coherence.

minor comments (5)

[§4.3] The paragraph beginning 'Second, concerning accuracy preservation...' is duplicated verbatim.
[Figures 5 and 6] The figure labels and captions appear garbled in the text (e.g., '/uni00000013...' sequences). Please ensure the PDF rendering is correct and the figures are legible.
[Theorem 3.2] The 'Semantic Specificity Hierarchy' is stated as a theorem but is essentially a construction/design property of the DAG; no proof is given. Either provide a proof or relabel it as a stated design invariant.
[§3.3] There is a sentence fragment: 'While latest temporal indexing techniques (Hou et al., 2024; Tian et al., 2024) for specific domains, they are not suitable...' The grammar should be corrected.
[Abstract] The abstract mentions 'LoCoMo Refined' and an 'order-of-magnitude latency advantage' on that benchmark, but LoCoMo Refined is not defined or discussed anywhere in the body. Please add a description and the corresponding results.

Circularity Check

0 steps flagged

No significant circularity identified: SwiftMem's headline results are direct benchmark measurements, and its only self-citation (Tian et al. 2024) is not load-bearing.

full rationale

The central empirical claims are self-contained. The three-tier index is implemented as explicit data structures and measured on LoCoMo/LongMemEval against a fixed GPT-4.1-mini judge; no parameter is fitted to a subset and then 'predicted' on a closely related quantity. The query-tag router selects tags by cosine similarity (Eq. 3-4), and retrieval accuracy is judged against gold answers, so the tag-defined subset does not by construction determine the reported LLM-judge scores. The temporal index's O(log N) range-query claim is a standard binary-search invariant over sorted timelines (Section 3.3.1), not an imported uniqueness result. Theorem 3.3 gives an explicit complexity formula, though it omits the cost of scanning episode sets attached to selected tags - a real evidence/soundness gap in the sub-linearity claim, but that is an unproven bound, not a circular derivation. The ablation (Section 4.4) shows a 14-point LLM-score improvement with unchanged 90.5% evidence recall; that is an uncontrolled confound, not a case of the output being equal to the input by construction. The only self-citation, Tian et al. 2024, appears in a related-work sentence saying earlier temporal indexing techniques 'are not suitable for the conversational memory retrieval in agentic scenarios'; it is not used to justify SwiftMem's design, complexity, or benchmark numbers. Baseline numbers marked with a dagger are imported from Nemori, which is external evidence, not circular. No load-bearing step reduces to its own inputs, so the correct score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The speed claim depends on k, Dmax, and consolidation thresholds being set appropriately, and on tag/routing quality, which is not independently validated. No new physical or theoretical entities are introduced; DAG-Tag, temporal index, and co-consolidation are software components.

free parameters (3)

k (number of selected tags per query) = not reported
Eq. (4) selects top-k tags; k controls the recall/latency tradeoff of DAG routing, but no value or sensitivity analysis is reported.
Dmax (maximum DAG expansion depth) = not reported
Hierarchical expansion in §3.2.2 and §3.2.3 expands selected tags to depth Dmax; this affects recall and query time but is never specified.
Co-consolidation scoring thresholds = not reported
§3.4 uses a scoring function balancing cluster cohesion, cluster size, and memory fragmentation; the thresholds or objective details are not given.

axioms (4)

domain assumption LLM-generated tags are accurate, complete, and stable enough that query-tag embedding routing finds all relevant episodes.
If a relevant episode lacks the matched tag, pruning to the tag subset loses it; the paper reports no tag-recall or routing-accuracy metric.
domain assumption Conversational memory queries exhibit temporal and semantic locality strong enough that subset retrieval does not hurt accuracy.
Section 1's two 'critical observations' motivate the design; the paper does not test queries without such locality beyond the standard benchmarks.
ad hoc to paper Baseline latency and accuracy numbers borrowed from Nemori are measured under comparable conditions.
Tables 1 and 2 reuse baseline numbers with †; different hardware, LLM versions, and API latencies can change search latency by orders of magnitude.
domain assumption LLM-judge scores with GPT-4.1-mini are a valid proxy for memory retrieval quality.
Accuracy claims rest on LLM-as-judge; no human agreement or calibration analysis is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 14796 in / 12168 out tokens · 113932 ms · 2026-08-03T10:51:39.158855+00:00 · methodology

0 comments

read the original abstract

Agentic memory systems have become critical for enabling LLM agents to maintain long-term context and retrieve relevant information efficiently. However, existing memory frameworks often perform query-agnostic retrieval over the full memory embedding space even when their storage layer is backed by efficient vector indexes such as HNSW. This full-scope retrieval path creates latency bottlenecks as memory grows, hindering real-time agent interactions. We propose SwiftMem, a query-aware agentic memory system that narrows retrieval to query-relevant memory subsets through specialized indexing over temporal and semantic dimensions. Our temporal index enables logarithmic-time range queries for time-sensitive retrieval, while the semantic DAG-Tag index maps queries to relevant topics through hierarchical tag structures. To address memory fragmentation during growth, we introduce an embedding-tag co-consolidation mechanism that reorganizes storage based on semantic clusters to improve locality. Across LoCoMo and LongMemEval$_S$, SwiftMem reaches 10.8/12.7 ms search latency while maintaining competitive LLM-judge accuracy against strong HNSW-backed memory systems. On the calibrated benchmark, LoCoMo Refined, SwiftMem remains close to the top LLM-judge score while preserving an order-of-magnitude latency advantage.

Figures

Figures reproduced from arXiv: 2601.08160 by Anxin Tian, Hui-Ling Zhen, Lei Chen, Mingxuan Yuan, Xianzhi Yu, Xing Li, Yiming Li, Zhenhua Dong.

**Figure 1.** Figure 1: Architectural comparison between existing memory frameworks and SwiftMem. Traditional approaches traverse the entire storage layer for each query (left), while SwiftMem employs query-aware indexing to fetch only relevant memory subsets (right). linear retrieval complexity through intelligent indexing over memory’s natural organizational dimensions. SwiftMem’s design philosophy is simple yet powerful: analy… view at source ↗

**Figure 2.** Figure 2: SwiftMem’s end-to-end query workflow. User queries are first analyzed by the Query-Tag Router and Temporal Indicator to determine relevant search dimensions. The three-tier indexing architecture (Temporal Index, DAG-Tag Index, and Embedding Index) then retrieves only the relevant memory subset from the storage layer, avoiding exhaustive scanning. cognitive science-inspired mechanisms including svent segme… view at source ↗

**Figure 3.** Figure 3: A Semantic DAG-based Tag Example between Andrew and Audrey on LoCoMo (Maharana et al., 2024). It organizes memories where pets serves as the root topic, with child concepts representing semantic aspects (dogs, companionship) and their derived emotions (happiness, comfort), enabling queries to match either specific terms or abstract themes. specific concepts level by level up to a specified depth parameter… view at source ↗

**Figure 4.** Figure 4: The Scalability Evaluation on LoCoMo Dataset Mem scales sub-linearly with conversational length as larger scales enable better memory consolidation. All measurements remain well below the 15ms threshold, confirming SwiftMem’s ability to deliver consistent real-time performance regardless of memory scale. Second, concerning accuracy preservation, SwiftMem exhibits robust performance stability with LLM sc… view at source ↗

**Figure 6.** Figure 6: The Ablation Study on Tag-Embedding Coconsolidation. relevant memory subsets and introducing embedding-tag co-consolidation to optimize storage organization, SwiftMem achieves 47× faster search compared to state-of-theart baselines while maintaining competitive accuracy on LoCoMo and LongMemEval benchmarks. These results demonstrate that query-aware indexing provides a practical path toward scalable, re… view at source ↗

discussion (0)

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Exploring Cross-Scenario Generality of Agentic Memory Systems: Diagnostics and a Strong Baseline
cs.AI 2026-06 unverdicted novelty 5.0

An agentic harness letting the LLM self-manage flat text-file storage via tool calls outperforms eight prior memory systems on cross-scenario generality across QA, chat, trajectory, stress-test, and long-horizon tasks.
Mitigating Provenance-Role Collapse in Long-Term Agents via Typed Memory Representation
cs.CL 2026-05 unverdicted novelty 5.0

MemIR is a typed memory representation for LLM agents that structures memory into atoms separating evidence, cues, and claims, leading to better performance on source tracking tasks in experiments on LoCoMo and BEAM-100K.

Reference graph

Works this paper leans on

2 extracted references · cited by 2 Pith papers

[1]

Extract 3-8 meaningful tags that capture the main topics, themes, and contexts
[2]

tags": [

Identify hierarchical relationships between these tags (parent-child) Guidelines for tags: - Tags should be lowercase, single words or short phrases (max 3 words) - Focus on: topics, activities, locations, entities, emotions, intents - Prioritize specific over generic (e.g., ‘python_programming’ over ‘technology’) - Use underscores for multi-word tags (e....

[1] [1]

Extract 3-8 meaningful tags that capture the main topics, themes, and contexts

[2] [2]

tags": [

Identify hierarchical relationships between these tags (parent-child) Guidelines for tags: - Tags should be lowercase, single words or short phrases (max 3 words) - Focus on: topics, activities, locations, entities, emotions, intents - Prioritize specific over generic (e.g., ‘python_programming’ over ‘technology’) - Use underscores for multi-word tags (e....