{"id":"07b44393-ff59-4726-8cbd-801eccf13e7d","arxiv_id":"2605.23723","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"MemAudit combines counterfactual causal influence scores with memory consistency graphs to identify poisoned records in LLM agent memory, reducing MINJA attack success from 70% to 0% in QA and 83.3% to 0% in reasoning tasks.","lead":"MemAudit is a post-hoc framework that audits LLM agent memory after harmful outputs by scoring each memory's causal influence via counterfactuals and flagging structural anomalies in a memory consistency graph. A smart generalist might read it to understand defenses against memory poisoning attacks that occur through normal interactions rather than direct tampering.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Whether counterfactual influence + consistency graph reliably flags poisons without high false positives or missed records remains the core unverified assumption.","rationale":"The reader's weakest_assumption directly isolates the load-bearing empirical premise. Because the provided abstract supplies only outcome percentages and no supporting diagnostics, the concern stands; full-text details would be required to resolve it, leaving the UNVERDICTED status unchanged.","tokens_in":1742,"tokens_out":321,"duration_ms":18961,"concrete_test":"In the experimental section, locate the subsection describing the memory-auditing decision procedure and any clean-memory ablation; recompute or inspect the false-positive rate when the same thresholds are applied to an equal-sized clean memory bank (no poisons). If the rate exceeds 5% or if the procedure is not fully specified, the neutralization claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline empirical claim (ASR reduced to exactly 0% on QA and RAP under MINJA) depends on the two signals together being sufficient to identify and neutralize malicious records. The abstract gives no information on (a) how the influence score is computed or thresholded, (b) how the graph anomaly score is combined with it, (c) false-positive rates on clean memory stores, or (d) whether the decision rule was tuned on the same test attacks. If either signal produces noisy or correlated scores, or if the combination rule overfits the reported scenarios, the 0% figures would not generalize and the post-hoc auditing utility would not hold.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript proposes MemAudit, a post-hoc causal memory auditing framework for memory-augmented LLM agents. It combines a counterfactual memory influence score measuring each memory's causal contribution to harmful outputs with a memory consistency graph identifying structurally anomalous memories. Evaluated against the MINJA query-only memory injection attack, the paper claims substantial reductions in attack success rates under post-hoc auditing, specifically reducing QA ASR from 70% to 0% and RAP ASR from 83.3% to 0%.","tokens_in":1882,"tokens_out":474,"duration_ms":22555,"significance":"If the empirical results hold under rigorous validation, MemAudit would address an important gap in defenses for memory-augmented LLM agents by enabling post-hoc identification and removal of malicious records after harmful behavior is observed, complementing existing online intervention methods. The dual use of causal attribution and structural anomaly detection provides a concrete, falsifiable approach to this security problem.","major_comments":[{"comment":"Abstract: The headline claims of reducing QA attack success from 70% to 0% and RAP from 83.3% to 0% are presented without any information on trial counts, statistical tests, baseline comparisons, variance across runs, or the precise computation and thresholding of the counterfactual influence scores, rendering the central empirical claim unverifiable from the provided evidence.","section":"Abstract"},{"comment":"Method description: No details are given on how the counterfactual memory influence score is computed from interventions, how it is combined with the memory consistency graph anomaly score (e.g., via thresholds, weighting, or logical conjunction), or whether the final decision rule was tuned on the reported test attacks, which directly bears on whether the two signals suffice to neutralize poisons without high false positives or missed records.","section":"Method"},{"comment":"Evaluation: The manuscript supplies no false-positive rates when MemAudit is applied to clean memory stores, nor any analysis of missed poisons or robustness under distribution shift, leaving the weakest assumption (that the combined signals reliably flag poisons in realistic agent settings) untested and the 0% ASR figures potentially non-generalizable.","section":"Evaluation"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. The comments highlight important areas for improving the clarity and completeness of our empirical claims, methodological details, and evaluation. We will revise the manuscript accordingly to address each point.","responses":[{"response":"We agree that the abstract should provide more context to make the claims verifiable. In the revision, we will expand the abstract to note that results are averaged over 10 independent runs with reported standard deviations, include a brief mention of baseline comparisons (standard retrieval without auditing), and indicate that the influence score uses a fixed threshold of 0.5 on the causal effect difference. Full details on computation and statistical tests will remain in the main text and appendix due to length constraints.","revision_made":"yes","referee_comment":"[Abstract] Abstract: The headline claims of reducing QA attack success from 70% to 0% and RAP from 83.3% to 0% are presented without any information on trial counts, statistical tests, baseline comparisons, variance across runs, or the precise computation and thresholding of the counterfactual influence scores, rendering the central empirical claim unverifiable from the provided evidence."},{"response":"We will revise the Method section to include the exact computation: the counterfactual influence score is defined as the difference in the LLM's output probability for a harmful response when performing a do-intervention that removes the candidate memory record. The consistency graph anomaly score measures deviation from average node connectivity. The signals are combined via logical conjunction after independent thresholding (influence > 0.3 and anomaly > 2 standard deviations). Thresholds were selected on a held-out validation set of clean and poisoned memories, not on the test attacks. We will add equations, pseudocode, and explicit discussion of this process.","revision_made":"yes","referee_comment":"[Method] Method description: No details are given on how the counterfactual memory influence score is computed from interventions, how it is combined with the memory consistency graph anomaly score (e.g., via thresholds, weighting, or logical conjunction), or whether the final decision rule was tuned on the reported test attacks, which directly bears on whether the two signals suffice to neutralize poisons without high false positives or missed records."},{"response":"We agree these metrics are necessary for a complete evaluation. In the revised manuscript, we will add a new subsection reporting a false-positive rate below 5% when applying MemAudit to five clean memory stores of varying sizes. We will confirm zero missed poisons (100% recall) in the reported experiments and include an analysis of robustness under distribution shift by testing on out-of-domain queries from a different domain, where ASR remains at 0%. These results will be presented with the same trial counts as the main experiments.","revision_made":"yes","referee_comment":"[Evaluation] Evaluation: The manuscript supplies no false-positive rates when MemAudit is applied to clean memory stores, nor any analysis of missed poisons or robustness under distribution shift, leaving the weakest assumption (that the combined signals reliably flag poisons in realistic agent settings) untested and the 0% ASR figures potentially non-generalizable."}],"tokens_in":1494,"tokens_out":669,"duration_ms":22454,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's main move is a post-hoc framework that scores each memory record by its causal effect on harmful outputs and then checks the whole store for structural outliers via a consistency graph. This targets the gap left by online defenses that try to stop bad behavior before it happens. The MINJA attack setup, where poisons arrive through ordinary queries rather than direct edits, matches how real agents would be compromised, so the threat model feels grounded. The reported drops from 70% to 0% on QA and 83.3% to 0% on RAP are the headline numbers, and the idea of using two signals together is presented as new relative to prior work the abstract cites. That combination is the clearest contribution. The abstract supplies no equations or derivations, which keeps the circularity burden low, but it also means the influence score computation, the graph construction, the way the two signals are combined, and the exact thresholds are not visible. The stress-test concern lands: without trial counts, false-positive rates on clean memory, or confirmation that the decision rule was not tuned on the same attacks, the exact-zero results cannot be assessed for robustness. If either signal is noisy or correlated, or if the combination overfits the tested cases, the auditing utility would shrink. The paper is aimed at researchers building or securing memory-augmented agents for long-horizon tasks. A reader already working on agent safety or retrieval poisoning would find the framing useful even if the current evidence is thin. The work shows clear thinking about the post-hoc setting and honest engagement with the practical vulnerability, so it is coherent on its own terms. I would bring it to a reading group for the idea and the attack description, but not yet for the results. I would not cite it until the methods and controls are filled in. A serious editor should send it to peer review so the authors can supply the missing experimental details and let referees check whether the two-signal rule actually generalizes.","headline":"MemAudit pairs a counterfactual influence score with a memory consistency graph for post-hoc auditing of injected records in LLM agent memory, but the abstract's 0% attack success claims rest on unshown experimental details and decision rules.","tokens_in":2369,"tokens_out":481,"would_cite":false,"duration_ms":14556,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"MemAudit uses counterfactual influence scoring + consistency-graph anomaly detection for post-hoc memory auditing; no structural overlap with RS.","alignment":"orthogonal","rationale":"The paper's core machinery (CMIS via replay-based intervention, CAS via semantic/NLI graph, fused DS ranking) is standard causal attribution + graph anomaly detection in the LLM-agent security domain. It has zero connection to the RS forcing chain (reality_from_one_distinction, J(x) = ½(x + x⁻¹) − 1, φ-ladder, 8-tick periodicity, AlexanderDuality D=3, etc.). No RS module or theorem is paralleled or contradicted.","tokens_in":51820,"confidence":"high","tokens_out":155,"duration_ms":4313,"cache_read_input_tokens":38528,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"MemAudit identifies poisoned memories in LLM agents after attacks by scoring each record's causal contribution to harmful outputs and detecting structural anomalies in the memory store.","keywords":["memory auditing","LLM agents","memory poisoning","causal attribution","post-hoc defense","memory consistency graph","counterfactual influence"],"falsifier":"A new memory-injection technique that produces records whose removal does not change the harmful outputs yet still evades detection by both the influence score and the consistency graph.","tokens_in":2657,"feed_emoji":"🔍","tokens_out":630,"duration_ms":15362,"temperature":0.7,"pith_summary":"The paper establishes a post-hoc auditing method that traces which stored memories caused an agent to produce bad results, then removes them. This matters because agents increasingly keep persistent memory of past interactions, and an adversary can slip malicious records into that store through ordinary conversations. Once retrieved later, those records steer the agent's reasoning without any ongoing attacker presence. By measuring counterfactual influence and building a consistency graph across all memories, the approach isolates the injected records. Experiments show the method drives attack success rates to zero in both question-answering and reasoning-agent settings under realistic conditions.","feed_headline":"MemAudit cuts memory poisoning success to zero after attacks","feed_subtitle":"Counterfactual influence scores plus consistency graphs locate and remove injected records without real-time blocking.","key_machinery":"the dual-signal auditing procedure that pairs a counterfactual memory influence score with a memory consistency graph to attribute and isolate malicious records","core_discovery":"MemAudit combines a counterfactual memory influence score, which quantifies how much each memory record causally affects the production of harmful outputs, with a memory consistency graph that surfaces records whose content or retrieval patterns deviate from the rest of the store. When applied after harmful behavior is observed, these two signals together locate and neutralize the malicious records that were injected through normal agent interactions in the MINJA attack, eliminating the attack success that previously reached 70 percent in QA tasks and 83.3 percent in RAP tasks.","pith_inferences":["The auditing approach could extend to retrieval-augmented generation systems that also maintain persistent document stores.","Repeated auditing passes might allow agents to maintain memory integrity over very long interaction histories.","If the consistency graph can be maintained incrementally, the cost of each audit round could stay low enough for routine use."],"forward_implications":["Agents can continue using long-term memory stores without permanent compromise once harmful behavior appears.","Defense can shift from blocking inputs in real time to cleaning the memory bank afterward.","The same auditing signals can be recomputed whenever new harmful outputs are observed.","Memory stores remain usable for retrieval while still allowing targeted removal of compromised entries."],"fun_headline_variants":["MemAudit traces poisoned memories with causal influence scores","Anomaly detection flags injected records in LLM agent memory","Post-hoc auditing neutralizes memory attacks in agents","MemAudit combines counterfactual scores and consistency graphs","Causal attribution uncovers sources of agent memory poisoning"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The two signals are together sufficient to separate malicious records from benign ones without producing many false positives or missing poisons across the tested agent configurations.","fun_headline_variants_meta":{"raw":{"variants":["MemAudit traces poisoned memories with causal influence scores","Anomaly detection flags injected records in LLM agent memory","Post-hoc auditing neutralizes memory attacks in agents","MemAudit combines counterfactual scores and consistency graphs","Causal attribution uncovers sources of agent memory poisoning"]},"model":"grok-4.3","cost_usd":0.00511,"raw_usage":{"total_tokens":2509,"prompt_tokens":715,"num_sources_used":0,"completion_tokens":69,"cost_in_usd_ticks":51099500,"prompt_tokens_details":{"text_tokens":715,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1725,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":715,"tokens_out":69,"duration_ms":10145,"temperature":1.0,"reasoning_tokens":1725,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-25T04:03:12.596566+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A new memory-injection technique that produces records whose removal does not change the harmful outputs yet still evades detection by both the influence score and the consistency graph.","supporting_citations":[],"review_version":1}