{"id":"11356deb-b734-44aa-b635-8c8eea686e6b","arxiv_id":"2608.12888","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ReFind, an agent-controlled keyword search over raw chat logs, outperforms structured memory systems such as HippoRAG 2 on conversational-memory benchmarks without building any semantic index.","lead":"A new agent system called ReFind answers questions from chat history by searching the raw, unmodified conversation log with keywords, instead of first building summaries or knowledge graphs. It beat graph-based and tree-based memory systems on several conversational memory benchmarks, suggesting much of the benefit of structured memory can come from better retrieval control.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Structured baselines lack the agentic search loop: on LongMemEval-M, one-search ReFind (68.9) is tied with HippoRAG 2 (66.7) while full ReFind is 89.3, so the raw-vs-structure claim is confounded with retrieval control.","rationale":"The reader's weakest assumption is baseline parity: most comparison numbers are reused from prior papers, so harness or judge mismatches could shift the margins. That is a real and transparently disclosed concern, and I agree it warrants the CONDITIONAL verdict. However, I see a more design-level threat to the causal claim. The paper asks how much benefit comes from structure itself versus competent retrieval over the raw history, but its comparison to structured memory systems does not hold retrieval control fixed. The strongest structured baseline, HippoRAG 2, is evaluated in its published pipeline, while ReFind includes a multi-round agent with note-taking and session-level state. The one-search ablation in Table 4 quantifies the consequence: on LongMemEval-M, ReFind without multi-round search scores 68.9, which is essentially HippoRAG 2's 66.7, while the full agent reaches 89.3. Thus the headline gap on M is dominated by the agentic loop, not by the choice of raw lexical records over a graph. Appendix C explicitly notes that systems were not rerun with a matched controller or tool-call budget, which is the crux. This does not invalidate the system-level empirical result that the full ReFind interface outperforms published structured systems; nor does it undermine the internal ablations, which cleanly separate the agent loop, the chat-native controls, and the lexical backend. But it does weaken the stronger conclusion that raw records and conversational structure, rather than adaptive control, are what outperform structured memory. The paper's own transparency and the strength of its controlled ablations justify keeping the verdict CONDITIONAL rather than moving to reject or accept.","tokens_in":17980,"tokens_out":7214,"duration_ms":74226,"concrete_test":"Run a positive control: take HippoRAG 2's graph index and drive it with the authors' exact four-iteration ReAct controller, note-taking, answer stage, and GPT-5-mini backbone on LongMemEval-S/M, adapting the four chat-native controls to graph retrieval (expansion to neighboring graph nodes, session provenance for context, timestamp filters, and seen-session exclusion). If this 'agentic HippoRAG 2' reaches or exceeds ReFind's 93.2/89.3, the paper's claim that raw logs outperform structured memory is not supported; if it stays near 80/66.7, the raw-record explanation is strengthened. Report per-question results on the 15 LongMemEval-M questions, since one answer changes accuracy by 6.7 points.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central attribution claim—that preserving raw records outperforms first transforming them into a memory representation—is not tested with the retrieval controller held fixed. In Tables 2 and 3, structured systems such as HippoRAG 2, GraphRAG, and RAPTOR appear in their published, non-agentic configurations, while ReFind is given a four-iteration ReAct loop with note-taking, context expansion, session-aware reranking, temporal filtering, and deduplication. The paper's own one-search control shows how much this matters: on LongMemEval-M, one-search ReFind scores 68.9, essentially tied with HippoRAG 2 (66.7), whereas the full four-iteration ReFind reaches 89.3. The 22.6-point LongMemEval-M gap is therefore mostly attributable to the multi-round agent, not to the raw-lexical representation. Appendix C acknowledges that 'systems were not all rerun with a matched controller or tool-call budget,' but that admission undercuts the headline conclusion as stated. Baseline reuse (Table 2 from Hu et al. 2025; Table 3 from STITCH, with only GAM re-run) is a related comparability problem, but even perfectly reproduced baseline numbers would leave this confound: a structured memory system equipped with the same adaptive search loop might match or beat ReFind.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ReFind, an agent-controlled retrieval interface that performs iterative BM25 keyword search directly over unmodified, session-structured chat logs, with no offline LLM-based index construction. The retrieval stage uses a ReAct-style controller with a search tool, note-taking, and four chat-native controls: session-aware rank fusion via RRF, local context expansion, temporal filtering, and seen-session deduplication; a separate reasoning stage answers from the collected evidence. The authors report mean accuracy 58.2 across six MemoryAgentBench tasks with GPT-4o-mini, above HippoRAG 2 (53.2), and 93.2/89.3 on LongMemEval-S/M with GPT-5-mini. Controlled ablations compare the full method with a generic agentic BM25 control, a one-search control, component removals, and dense/hybrid backends. The paper concludes that preserving raw records and exposing their conversational structure to an adaptive agent outperforms transforming them into a structured memory representation.","tokens_in":18259,"tokens_out":3843,"duration_ms":37313,"significance":"If the central claim holds, this is a valuable negative result for the agent-memory literature: much of the benefit attributed to pre-built summaries, embeddings, trees, and knowledge graphs may be recoverable with a competent retrieval controller over the raw archive. The paper has genuine strengths: the ablation design is unusually disciplined, with a matched generic-agentic control, a one-search control, component removals, and backend swaps that separate iteration, chat-native controls, and lexical retrieval; hyperparameters such as BM25 k1, b, RRF k, and top-K are stated and are standard defaults rather than test-set-tuned values; repeated runs on the LongMemEval subsets quantify execution variability; and the example trajectory in Appendix G makes the mechanism auditable. The main limitation is comparability of baselines, since the headline comparisons reuse numbers from prior papers and do not hold the agentic controller fixed across the raw-versus-structured contrast.","major_comments":[{"comment":"The paper's central attribution claim—that preserving raw records outperforms first transforming them into a memory representation—is not tested with the retrieval controller held fixed. On LongMemEval-M, the one-search ReFind control scores 68.9, essentially tied with HippoRAG 2's 66.7 in Table 3, while the full four-iteration ReFind reaches 89.3; the 22.6-point gap over HippoRAG 2 is therefore mostly attributable to the multi-round agent loop rather than to the raw lexical representation itself. Because the structured baselines in Tables 2 and 3 appear in their published non-agentic configurations while ReFind uses a four-iteration ReAct controller with note-taking, context expansion, temporal filtering, and deduplication, the raw-versus-structure contrast is confounded with the presence of an adaptive retrieval controller. I request a matched-controller experiment, such as running HippoRAG 2 or GraphRAG behind the same four-iteration ReAct loop over their indices with the same tool-call budget, or an explicit re-scoping of the conclusion to agent-controlled search over raw logs versus non-agentic structured memory.","section":"§5, Table 4; §3 System Architecture"},{"comment":"The headline comparisons reuse baseline numbers from Hu et al. (2025) for MemoryAgentBench and from STITCH (Yang et al., 2026) for LongMemEval, with only GAM re-evaluated locally, and Appendix C acknowledges that the systems were not all rerun with a matched controller or tool-call budget. Because the MemoryAgentBench margin is small (58.2 vs. 53.2, and 51.3 vs. 50.7 on LME) and the LongMemEval baselines come from a different evaluation harness, unmeasured differences in judge prompts, decoding settings, or incremental-turn formatting could change the ordering. I ask the authors to reproduce at least the strongest competitor in each table under their local harness and protocol, or to report per-question agreement between the reused and locally evaluated settings so that the magnitude of the comparison can be verified.","section":"§4 Baselines, Tables 2 and 3; Appendix C"},{"comment":"The LongMemEval-M subset contains only 15 questions, so a single changed judgment moves accuracy by 6.7 points, and the five-run full-method mean on M is 89.3 ± 6.0 with run-level values 80.0–93.3. The component-ablation deltas on M (e.g., −9.3 for session deduplication, −4.9 for context window) fall within this run-to-run dispersion, and the appendix itself cautions that the M ordering requires caution; the main text nevertheless presents these deltas as evidence for the individual contribution of each control. I request paired or hypothesis-test-based reporting for the M subset, or an explicit main-text caveat that the M-level component ordering is descriptive and not statistically distinguished from noise.","section":"Table 4; Appendix C, Repeated Component Ablations"}],"minor_comments":[{"comment":"The full-text title on page 1 contains a rendering artifact, 'When Y our Agent Opens the Chat App,' which should be corrected to match the abstract title.","section":"Title and Abstract"},{"comment":"The appendix reproduces the ReFind prompts and the LongMemEval judge prompt, but the MABench LME judge protocol is described only as 'task-specific templates from the MABench codebase'; including those templates or linking them explicitly would make the evaluation fully reproducible.","section":"Appendix A"},{"comment":"Resource-use reporting would be more useful for practical claims if monetary cost per question were included, since the paper emphasizes that ReFind avoids offline index construction but does not report API cost for the online loop.","section":"Appendix D, Table 11"},{"comment":"The table's note that deltas are 'descriptive, not paired' is helpful, but the main-text sentence 'the resulting evidence chain attributes the gain to the interaction between multi-round control and a conversationally structured lexical interface' should be softened to reflect that the one-search and component comparisons are unpaired means over different run counts.","section":"§5, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The baseline-reuse issue is acknowledged honestly in Appendix C, but it is load-bearing for the headline, and the one-search control in Table 4 shows that the controller, not just the representation, drives the LongMemEval-M margin. If the authors can rerun at least one structured memory system under the matched agentic controller and reproduce the strongest reused baselines locally, the paper could become a strong contribution. I do not see a circularity or novelty-disclosure problem: the self-cited A-RAG appears only as related work, and the contributions are framed as empirical results rather than derivations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper builds a real system and evaluates it more carefully than most of this subfield, but the headline conclusion as stated outruns the data. What's genuinely new: ReFind, a no-structure agentic lexical search over raw chat logs, with four chat-native controls, and a clean ablation chain that separates multi-round control, component contributions, and retrieval backend. The matched generic-agentic BM25 control is a good idea, and the repeated runs on LongMemEval are better practice than the field usually sees. The appendix is transparent about baseline reuse and the small M sample.\n\nThe soft spot is the central attribution claim. Structured baselines are not given the same agentic search loop. The paper's own one-search control makes the problem visible: on LongMemEval-M, one-search ReFind is 68.9, essentially tied with HippoRAG 2 at 66.7, while full ReFind is 89.3. So most of that 22.6-point margin is the multi-round ReAct loop, not the raw-lexical representation. The generic-agentic BM25 control at 82.2 on M shows the chat-native controls add 7.1 points, but that still does not tell you whether HippoRAG 2 with a matched agent loop would match or beat ReFind. Appendix C acknowledges that systems were not rerun with a matched controller, but the abstract and conclusion draw the stronger inference anyway.\n\nOther issues are minor and mostly flagged: Table 2 and Table 3 reuse baseline numbers from prior papers with only GAM rerun; M has 15 questions, so one changed judgment moves the score 6.7 points; and there is no human-agreement study for the judge. None of these are fatal.\n\nWho is this for: anyone building agent memory systems. The interface is cheap, auditable, and append-only, which is practically valuable. The paper deserves a serious referee. I would accept with revision: either rerun the key structured baselines with a matched agentic loop, or reframe the conclusion as 'agentic lexical search is competitive and much cheaper' without claiming structure is unnecessary.","headline":"Real system, careful ablations, but the 'raw beats structure' conclusion is confounded by the agentic loop—reframe or rerun.","tokens_in":18815,"tokens_out":3642,"would_cite":true,"duration_ms":32124,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An agent-controlled lexical search over unmodified chat logs can outperform structured memory systems on conversational-memory benchmarks, without any LLM-based index construction.","keywords":["conversational memory","agentic retrieval","BM25","raw chat logs","structured memory","Reciprocal Rank Fusion","MemoryAgentBench","LongMemEval"],"falsifier":"If a direct reproduction of the six MemoryAgentBench baselines—HippoRAG 2, GraphRAG, RAPTOR, Mem0, Zep, Self-RAG, MemGPT, MIRIX, and BM25-RAG—under the paper's exact harness, judge prompts, and decoding settings with GPT-4o-mini produced a mean accuracy at or above ReFind's 58.2, or if rerunning LongMemEval-S/M baselines locally narrowed the 93.2/89.3 margin to within a few points, the headline ranking would collapse.","tokens_in":17763,"feed_emoji":"💬","tokens_out":6178,"duration_ms":52505,"temperature":0.7,"pith_summary":"ReFind asks how much of the benefit credited to structured agent memory comes from pre-built summaries, embeddings, trees, or knowledge graphs, and how much simply from competent retrieval over the original conversation history. Its answer is to build no semantic structure at all: leave the chat log unmodified, index it lexically at turn granularity, and let a language-model agent run iterative keyword searches with four chat-aware controls—retrieving nearby turns, boosting sessions with multiple hits, restricting by date, and skipping sessions already inspected. Across six conversational-memory tasks totaling about 2,800 questions, this reaches a mean accuracy of 58.2, above the strongest compared structured system, HippoRAG 2, at 53.2 on a matched GPT-4o-mini backbone, with no LLM-based index construction. The result matters because it suggests that much of the value attributed to elaborate memory representations is recoverable by preserving raw records and giving the agent controllable search.","feed_headline":"Searching raw chat logs beats prebuilt memory in agent recall tests","feed_subtitle":"Iterative keyword search hit 58.2 percent mean accuracy, above HippoRAG 2's 53.2, with no built index.","key_machinery":"The central mechanism is an agent-controlled search interface over an unmodified chat archive: a BM25 inverted index built at turn granularity with no LLM-generated representation, a ReAct-style controller that issues up to four parameterized keyword searches, saves evidence after each round, and conditions its next query on returned observations, and four chat-native controls—Reciprocal Rank Fusion (RRF) combining turn-level and session-level scores, expansion to ±2 neighboring turns around a hit, timestamp-range filtering, and session deduplication that excludes sessions already returned. A separate reasoning stage consumes the saved notes, grouped and chronologically ordered, and generates the answer from that evidence-only view.","core_discovery":"The paper's central claim is that for precise, evidence-grounded questions over chat archives, preserving raw records and exposing their conversational structure to an adaptive agent outperforms first transforming them into a memory representation. Concretely, ReFind reaches 93.2±3.3 on LongMemEval-S and 89.3±6.0 on LongMemEval-M with a GPT-5-mini controller, and on the six MemoryAgentBench tasks its 58.2 mean accuracy exceeds every compared baseline, including HippoRAG 2's 53.2. The paper further argues, through three matched control families, that the gain comes from the interaction between multi-round agent control and a session-, context-, time-, and redundancy-aware lexical interface, not from a larger model, semantic embeddings, or a single strong first query.","pith_inferences":["A testable extension beyond this paper: build a paraphrase-heavy variant of these benchmarks where questions share few surface terms with the evidence; ReFind's margin should narrow because the mechanism depends on discriminative lexical forms.","An implicit design consequence: production systems could layer a small structured index over ReFind-style raw search for latency-sensitive or abstractive workloads, since the paper's contribution is access intelligence, not a claim that no structure is ever useful.","The same session-aware controls should transfer to other timestamped, threaded corpora, such as email archives mapped to sessions by thread and sender, where the refinding studies cited in the paper already observed similar orienteering behavior.","Because the LongMemEval-M subset contains only 15 questions, fine-grained per-control rankings there are not stable; a larger question sample would settle the per-component contributions."],"forward_implications":["The paper shows that elaborate pre-built memory structures are not necessary for precise, evidence-grounded conversational-memory questions; raw records plus adaptive lexical search can outperform them on this task suite.","Because no LLM-based index is built, an archive becomes searchable as soon as messages arrive, and computation happens only at question time, averaging about 2.5 searches and 5 LLM calls per question.","Each of the four chat-native controls matters: removing any one costs 1 to 9 points on LongMemEval, with context expansion the clearest contribution on the shorter subset and session deduplication the largest on the longer subset.","The advantage persists under a stronger controller: on LongMemEval-S/M with GPT-5-mini, ReFind outperforms every compared structured system, and matched backend ablations show dense or hybrid retrieval do not beat lexical BM25, indicating the agent supplies semantic adaptation through reformulation.","A modular design principle follows: begin with faithful storage and a controllable search interface, then add derived structures only for workloads that demand a separate latency or abstraction layer."],"supporting_citations":[{"why":"Supplies the MemoryAgentBench incremental multi-turn task suite and the baseline numbers reused in the six-benchmark comparison.","marker":"(Hu et al., 2025)"},{"why":"Supplies the LongMemEval-S/M subsets, the evaluation protocol, and the GPT-5-mini baselines reused in the backbone-scaling comparison.","marker":"(Yang et al., 2026)"},{"why":"Provides HippoRAG 2, the strongest structured baseline the paper must beat.","marker":"(Gutiérrez et al., 2025)"},{"why":"Defines BM25, the lexical retrieval foundation of the search engine.","marker":"(Robertson and Zaragoza, 2009)"},{"why":"Defines Reciprocal Rank Fusion, the session-aware reranking mechanism.","marker":"(Cormack et al., 2009)"},{"why":"Provides the ReAct framework for the iterative evidence-collection agent loop.","marker":"(Yao et al., 2022)"},{"why":"GAM is the agentic raw-record baseline that the authors re-evaluate locally as the closest design comparison.","marker":"(Yan et al., 2025)"},{"why":"Defines LongMemEval, the benchmark whose S/M subsets measure long-term conversational memory.","marker":"(Wu et al., 2024)"},{"why":"Empirical refinding study motivating multi-step 'orienteering' search behavior that grounds the design of the agent controls.","marker":"(Teevan et al., 2004)"}],"fun_headline_variants":["Raw chat search beats built memory: ReFind at 58.2%","No index, no problem: agentic raw-log search tops memory graphs","ReFind: unmodified chat logs outdo structured memory","Agent-controlled raw-log search outperforms graph memory","Raw chat agent hits 58.2% beating HippoRAG 2's 53.2%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Most comparison numbers are reused from prior papers rather than measured in this harness, so the reported lead depends on those original evaluations being directly comparable to the local runs.","fun_headline_variants_meta":{"raw":{"variants":["Raw chat search beats built memory: ReFind at 58.2%","No index, no problem: agentic raw-log search tops memory graphs","ReFind: unmodified chat logs outdo structured memory","Agent-controlled raw-log search outperforms graph memory","Raw chat agent hits 58.2% beating HippoRAG 2's 53.2%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00101,"raw_usage":{"total_tokens":4335,"prompt_tokens":1082,"completion_tokens":3253,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":698,"completion_tokens_details":{"reasoning_tokens":3156}},"tokens_in":698,"tokens_out":3253,"duration_ms":24669,"temperature":1.0,"reasoning_tokens":3156,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:14:58.290846+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a direct reproduction of the six MemoryAgentBench baselines—HippoRAG 2, GraphRAG, RAPTOR, Mem0, Zep, Self-RAG, MemGPT, MIRIX, and BM25-RAG—under the paper's exact harness, judge prompts, and decoding settings with GPT-4o-mini produced a mean accuracy at or above ReFind's 58.2, or if rerunning LongMemEval-S/M baselines locally narrowed the 93.2/89.3 margin to within a few points, the headline ranking would collapse.","supporting_citations":[{"cited_title":"Ackerman, and David R","cited_arxiv_id":null,"evidence_quote":"Empirical refinding study motivating multi-step 'orienteering' search behavior that grounds the design of the agent controls."}],"review_version":1}