{"id":"c9d733cb-f30b-450e-a2bc-bd85f19f284a","arxiv_id":"2505.19436","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A graph-based memory controller with LLM-powered intent classification reportedly eliminates hallucinations and confusions across 27 scripted multi-turn agent turns in four hand-written scenarios.","lead":"This paper proposes a memory system, the Task Memory Engine, that lets LLM agents track multi-step tasks as a graph instead of a flat chat history. The authors report that it removes all hallucinations and misinterpretations in four scripted test scenarios, but the evidence is small and partly in-sample.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TRIM few-shot prompt leaks exact test turns; zero-error result likely reflects memorization, not generalization.","rationale":"The reader identified the TRIM prompt contamination as the weakest assumption; I agree. This is the most load-bearing concern because the entire contribution is the zero-error result, and the result depends on TRIM's intent classifications. A single contaminated few-shot example can trivially produce the correct classification for a test turn that is verbatim in the prompt, so the zero-error count cannot be interpreted as evidence of generalization. The post hoc cart adaptation in Section 5.4 further undermines the claim that the system was not tuned on the evaluation scenarios. While the paper releases code and uses a fixed temperature, no repeated runs, no error bars, and no held-out data are reported. The abstract's 66.7%/83.3% figures also contradict Table 1's 100% reduction, but that inconsistency is secondary to the leakage issue. Because the central empirical claim is not supported as written, the reader's REJECT verdict remains appropriate; no verdict change is needed.","tokens_in":17187,"tokens_out":4060,"duration_ms":78227,"concrete_test":"Run the four scenarios again with the Appendix D few-shot prompt modified to exclude the three verbatim test turns (Cooking Round 4, Trip Round 9, Meeting Round 1), replacing them with paraphrases or different-domain examples. If zero errors persist over all 27 turns, the leakage concern is resolved; if any error appears, the reported zero-error claim is contaminated and a properly held-out benchmark (e.g., new scripts or a standard multi-turn dataset) is required before the claim can stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (0 hallucinations, 0 confusions over 27 turns; Table 1) is obtained from one scripted run. Its validity rests on TRIM's intent classifier actually classifying held-out turns. However, Appendix D presents the TRIM few-shot prompt with three verbatim test utterances embedded as examples: Cooking Round 4 ('Wait! There's no celery in the refrigerator at all...'), Trip Round 9 ('start: By the way, wasn't I departing from Boston?'), and Meeting Round 1 ('Schedule a team meeting on Thursday at 2 PM with Alice, Bob, and Carol.'). These exact inputs appear in the evaluation scripts (Tables 8, 7, 9 respectively). The paper does not state that these examples were excluded from the evaluation turns. If they were not, TRIM can output the labeled structured JSON for those turns by copying the example rather than by performing the decomposition/intent inference the method claims. Since the zero-error count includes these turns, the 100% reduction claim would not be established for novel inputs. Separately, Section 5.4 reports that the cart scenario first produced 1 hallucination and 1 confusion, and only achieved zero after a task-specific TRIM 'flattening' adaptation; this is post hoc tuning on the test scenario. Combined, these issues mean the central empirical claim is not yet supported by a valid held-out evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Task Memory Engine (TME), a memory controller that replaces linear conversation context with a structured task graph (tree or DAG) and uses a Task Representation and Intent Management (TRIM) module to classify user intents and update the graph. The authors claim that TME-DAG, without fine-tuning, eliminates hallucinations and misinterpretations across four scripted multi-turn tasks (27 user turns), outperforming ReAct, and yields token savings. The paper includes qualitative case studies, aggregated error counts, an ablation, a token-efficiency comparison, and open-source code and scripts.","tokens_in":17428,"tokens_out":4895,"duration_ms":43642,"significance":"If the central empirical claim were sound, TME would be a useful practical contribution: a modular, fine-tuning-free memory layer that makes off-the-shelf LLMs more robust to revisions and dependency changes in multi-turn interaction. The idea of encoding task state as a dependency graph and separating intent classification from response generation is reasonable and worth exploring. The paper also deserves credit for releasing code, benchmarks, and detailed interaction scripts, which would aid reproducibility if the evaluation issues were resolved. However, the current evidence does not support the headline '100% reduction' claim: the evaluation design leaks exact test inputs into the few-shot prompt, the cart scenario was adapted after seeing the test outcome, and multiple reported numbers are internally inconsistent. These are load-bearing problems, not cosmetic ones.","major_comments":[{"comment":"The TRIM few-shot prompt in Appendix D contains verbatim user utterances from the evaluation set: Cooking Round 4 ('Wait! There's no celery in the refrigerator at all...'), Trip Round 9 ('start: By the way, wasn't I departing from Boston?'), and Meeting Round 1 ('Schedule a team meeting on Thursday at 2 PM with Alice, Bob, and Carol.'). These exact turns are part of the 27-turn evaluation in Tables 7, 8, and 9. The paper never states that these examples were excluded from the evaluation turns. Because TRIM's few-shot classifier can copy the labeled output for these turns, the zero-error result for those turns does not demonstrate generalization. The 100% reduction claim is therefore not established for a held-out test set.","section":"Appendix D vs. Tables 7-9"},{"comment":"The cart-editing scenario initially produced 1 hallucination and 1 confusion under TME-DAG, and only achieved zero after the authors introduced a 'task-specific TRIM adaptation' that flattens the graph structure. This adaptation was introduced after observing the failure on this exact scenario, which is post-hoc tuning on the test set. The reported zero-error performance for the cart task in Table 1 reflects the adapted system, not the method as specified in Section 4.2. This makes the claim that TME-DAG achieves zero errors across all four tasks a statement about a scenario-specific modified system, not about the proposed general method.","section":"Section 5.4, Table 1"},{"comment":"The abstract states that TME 'reduces hallucinations by 66.7% and misinterpretations by 83.3% across 27 user turns,' while Section 6.1 and Table 1 report 0 vs. 3 hallucinations and 0 vs. 5 confusions relative to ReAct, which are 100% reductions. The abstract also says 'eliminates 100% of hallucinations and misinterpretations in three tasks,' but Table 1 shows zero errors in all four tasks. These numerical claims are irreconcilable. The headline quantitative result is internally inconsistent and must be corrected.","section":"Abstract vs. Section 6.1 / Table 1"},{"comment":"The formal description defines f_TRIM as 'an embedding-based classifier' and states 'z_j = f_TRIM(s_j, G)' with f_TRIM mapping to {new, update, check}. However, Section 4.2 explicitly states that the embedding-based classifier was abandoned in favor of an LLM-based few-shot approach. The formal model therefore does not describe the implemented system. The paper should either revise the formal description to reflect the LLM-based classifier or clearly distinguish the intended future formalization from the implemented prototype.","section":"Section 4.5, Eq. (2)"},{"comment":"All performance results are based on a single scripted run of each scenario, with no multiple trials, no variance reporting, and no statistical significance testing. The 27 turns are the same author-written scripts used in the case studies, and the error labels are assigned by the same researcher who wrote the scripts. Given the small sample size and the absence of any held-out or diversified evaluation, the strong claim of a '100% reduction' is not robustly supported even setting aside the leakage and post-hoc adaptation concerns. A proper evaluation with multiple runs, varied prompts, and independent labeling is needed.","section":"Section 6, Tables 1 and 3"}],"minor_comments":[{"comment":"The trip planning scenario is described as having 10 rounds in Section 5.1, but Table 7 contains 11 rounds. The total of all four scenarios would then be 28, not 27, as stated in Table 1. The round counts should be reconciled.","section":"Appendix B.1 vs. Section 5.1"},{"comment":"The terms 'misinterpretation' and 'confusion' are used interchangeably (e.g., abstract vs. Table 1). One term should be used consistently, and its operational definition should be stated.","section":"Throughout"},{"comment":"The token-efficiency result is demonstrated only on a separate form-filling task, not on the four main scenarios. The paper should clarify that the 19.4% savings applies to that specific task and not claim general token efficiency without further evidence.","section":"Section 6.2"},{"comment":"The conclusion refers to 'a dynamic, graph-based memory tree,' but the methodology consistently describes a DAG. The terminology should be aligned.","section":"Section 7"},{"comment":"The cart-editing script notes say the responses are 'based on terminal logs,' but the provenance of the responses is otherwise described as author-written case studies. The paper should state clearly how the responses were collected (e.g., actual system runs) and whether the scripts were fixed before or after observing system outputs.","section":"Appendix B.4"}],"recommendation":"major_revision","confidential_remarks":"The open-source release and detailed appendices are commendable, but the evaluation integrity is the central concern. The Appendix D leakage and the post-hoc cart adaptation are serious enough that the current version should not be accepted. However, both issues are addressable in principle: the few-shot examples can be replaced with held-out examples and the pre-adaptation results can be reported transparently. The abstract inconsistency is a simple fix. The single-run evaluation is a deeper limitation. If the authors can provide a clean held-out evaluation with multiple runs and consistent reporting, the paper may become publishable. I would not reject outright because the core idea is not disproven; it is simply unsubstantiated as presented."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Ye Ye has put together a sensible engineering idea: replace linear context with a DAG of task nodes and use an LLM-based intent classifier (TRIM) to decide whether each user turn is new, an update, or a check. That combination is not in the cited prior work, and the modular packaging—no fine-tuning, drop-in for off-the-shelf LLMs—is genuinely useful for people building agents. The code is on GitHub, the architecture diagrams are clear, and the failure modes of ReAct in the case studies are well described.\n\nWhere it falls apart is the evaluation. The headline claim of zero hallucinations and confusions over 27 turns is built on a single scripted pass with author-assigned error labels. That alone would be thin but not fatal. The fatal part is Appendix D: the TRIM few-shot prompt contains three verbatim user utterances from the evaluation turns (Cooking Round 4, Trip Round 9, Meeting Round 1). The paper never states that those turns were excluded from evaluation. If they weren't, TRIM can copy the labeled JSON from the example rather than classifying novel input. The zero-error count includes these turns, so the 100% reduction claim is not actually about generalization. That's a load-bearing leak, not a cosmetic issue.\n\nThe cart scenario makes things worse. Section 5.4 admits TME-DAG initially produced one hallucination and one confusion, and only reached zero after a task-specific TRIM 'flattening' adaptation introduced after seeing the test scenario. That is post hoc tuning on the test set. The abstract also contradicts Table 1: it reports 66.7%/83.3% reductions across 27 turns, while Table 1 shows 0 vs 3 and 0 vs 5, which are 100% reductions. The paper can't have it both ways.\n\nThe method itself is not bad. The DAG update rules, the replace operation, and the role of history tracking are clearly motivated. The token efficiency numbers come from a single 6-turn form-filling task and are not the point. Ablations show the components matter, but they inherit the same evaluation problem.\n\nWho is this for? Practitioners who want a concrete memory layer to try in their own agent prototypes, and researchers interested in evaluation methodology for LLM agents. The paper deserves a serious referee because the idea is plausible and the code is public, but it needs major revision: a held-out benchmark, multiple runs, pre-registered labels, and either removal of leaked examples or an explicit statement that they were excluded. As written, the central empirical claim doesn't hold.","headline":"A plausible memory-module design, but the zero-hallucination claim rests on a leaked few-shot prompt and a single scripted pass; the idea is worth a revision, not acceptance.","tokens_in":17958,"tokens_out":2381,"would_cite":false,"duration_ms":19881,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing flat context with a DAG-structured task memory eliminates hallucinations and confusions in LLM agents across scripted multi-step scenarios.","keywords":["task memory engine","TMS-DAG","spatial memory","intent classification","multi-step LLM agents","hallucination reduction","revision tracking","ReAct baseline"],"falsifier":"Re-run the four scripts with the TRIM prompt's few-shot examples replaced by unrelated examples of new, update, and check intents, and count hallucinations and confusions; if errors appear, the original zero-error claim depends on the test turns being present in the prompt.","tokens_in":16946,"feed_emoji":"🧠","tokens_out":5236,"duration_ms":34792,"temperature":0.7,"pith_summary":"The paper argues that linear context is the root cause of LLM agents' multi-turn failures and proposes replacing it with a spatial, graph-based memory. The Task Memory Engine (TME) maintains a directed acyclic graph of subtasks and dependencies, and its TRIM component classifies each turn as a new task, an update, or a query. Across 27 scripted turns in trip planning, cooking, meeting scheduling, and shopping cart editing, TME-DAG reports zero hallucinations and zero confusions, while ReAct reports 3 and 5. If the result holds, it means off-the-shelf LLMs can become revision-aware and dependency-consistent without fine-tuning, simply by changing how context is stored and retrieved.","feed_headline":"Graph memory cuts LLM agent errors to zero in scripted tasks","feed_subtitle":"Replacing flat context with a DAG-based memory removes all hallucinations and confusions across 27 multi-turn test turns.","key_machinery":"The load-bearing pieces are the Task Memory Structure (TMS-DAG) and the Task Representation and Intent Management (TRIM) module. TMS-DAG is a directed acyclic graph $G=(V,E)$ in which nodes hold subtask state (slot, value, history, parent, dependencies) and edges encode ordering or dependency constraints; it supports add, replace, delete, rollback, and dependency propagation. TRIM is a few-shot LLM-based classifier that decomposes user input into subtasks, assigns each an intent in {new, update, check}, infers relations such as replaces, and selects a minimal subgraph for the LLM prompt. The five-step workflow—decompose, classify, update graph, retrieve subgraph, generate—is what transforms a flat-context LLM into a revision-aware agent.","core_discovery":"The central claim is that replacing flat conversation history with a DAG-structured task memory eliminates cross-turn contradictions in multi-step LLM agents. TME-DAG stores each subtask as a node with a slot, value, history, parent, and dependency edges, so a user revision such as \"use mushrooms instead of celery\" propagates globally to every dependent subtask. TRIM, a few-shot LLM-based intent classifier, decomposes each input into subtasks, labels intents as new/update/check, and retrieves a compact subgraph $G'_i$ to pass to the LLM. In the four reported case studies, this pipeline achieves zero hallucinations and confusions across all 27 turns, achieves full task consistency in all four scenarios, and saves 19.4% tokens relative to a flat baseline in a form-filling task. The paper also reports that the shopping cart scenario initially produced one hallucination and one confusion until a task-specific flattened TRIM adaptation was applied, after which it matched ReAct.","pith_inferences":["If the few-shot examples in Appendix D are not held out, the zero-error numbers may largely reflect pattern matching to those exact turns; a held-out evaluation would show how much generalization the classifier actually has.","The approach is likely to shine on tasks with shared subtasks and global substitutions, while flat, sequential tasks may not need a DAG at all—consistent with the cart-editing result.","The token savings should grow with conversation length in tasks with many independent branches, but the LLM call to TRIM adds latency and cost that the paper does not fully account for.","A natural next test is to apply TME-DAG to longer, open-ended conversations with hundreds of turns, where current linear-context methods hit context windows."],"forward_implications":["A memory layer alone, with no fine-tuning, can carry global revisions across dependent subtasks.","Token cost per turn can stay bounded as conversations grow, because only the relevant subgraph is sent to the LLM.","The same architecture can be adapted to enterprise workflows where tasks share resources and revisions propagate.","Agents can answer historical queries correctly by tracking node histories instead of re-reading the transcript.","The open-source release allows other teams to test the same claims on new scripts and domains."],"supporting_citations":[{"why":"Supplies the human spatial-memory motivation for structured, non-linear task memory.","marker":"[1]"},{"why":"Defines chain-of-thought prompting, a linear baseline TME is compared against.","marker":"[8]"},{"why":"Defines ReAct, the main flat-context baseline whose hallucinations and confusions TME-DAG is claimed to eliminate.","marker":"[9]"},{"why":"Provides tree-based prompting alternative that lacks persistent state tracking.","marker":"[10]"},{"why":"Represents memory-augmented LLM approach that requires fine-tuning, contrast for TME's no-fine-tuning design.","marker":"[11]"},{"why":"Represents retrieval-augmented generation, an alternative memory approach without structured dependency tracking.","marker":"[5]"}],"fun_headline_variants":["Graph memory removes all agent errors in 27 turns","DAG-based memory cuts LLM agent mistakes to zero","Spatial memory makes LLM agents reliable across turns","Task memory engine: zero hallucinations, no fine-tuning","Graph structure erases multi-step agent hallucinations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the TRIM intent classifier's zero-error performance reflects generalization rather than memorization of the specific test turns, since the paper's few-shot prompt examples come from the same scripts being evaluated.","fun_headline_variants_meta":{"raw":{"variants":["Graph memory removes all agent errors in 27 turns","DAG-based memory cuts LLM agent mistakes to zero","Spatial memory makes LLM agents reliable across turns","Task memory engine: zero hallucinations, no fine-tuning","Graph structure erases multi-step agent hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000496,"raw_usage":{"total_tokens":2480,"prompt_tokens":1041,"completion_tokens":1439,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":1363}},"tokens_in":657,"tokens_out":1439,"duration_ms":7307,"temperature":1.0,"reasoning_tokens":1363,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:13:44.913989+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the four scripts with the TRIM prompt's few-shot examples replaced by unrelated examples of new, update, and check intents, and count hallucinations and confusions; if errors appear, the original zero-error claim depends on the test turns being present in the prompt.","supporting_citations":[],"review_version":1}