{"id":"71da9dae-f53d-4036-bd1e-4007e96ff46e","arxiv_id":"2505.05177","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"MARK proposes persistent refined-memory agents and a weighted relevance formula, and reports improved MedMCQA chatbot scores in a small, likely overlapping train-and-test setup.","lead":"MARK adds three specialized memory agents to an LLM chatbot so it can store domain knowledge and user corrections without retraining. The paper reports accuracy gains on a medical QA dataset, but the evaluation does not establish that the gains come from the framework rather than from retrieving the answer key.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Experiment 3 evaluates on the same 46 records used to build memory; the paper's own percentages (50.8% / 67.4%) are exactly the effect of flipping 31 of those 46, so the reported gains likely reflect retrieval of stored SME corrections, not generalization.","rationale":"I read the strongest claim as the empirical claim from Experiment 3: injecting structured memory built by MBS raises response accuracy on MedMCQA without fine-tuning. The load-bearing condition is that the 107 test records are independent of the 46 records whose SME corrections are stored in memory. The paper does not assert this independence; rather, the numbers in Sections 3.1.3 and 4 show the 46 records are a subset of the 107. The increase of 50.8% in correct responses and decrease of 67.4% in errors are exactly 31/61 and 31/46, so 31 of the originally incorrect records flipped to correct, leaving 15 incorrect in Experiment 4. That is the signature of retrieving stored corrections for the very records used to build memory. Under pure-similarity retrieval (c=1), the stored SME explanation for a query is the nearest memory, so the experiment measures memorization. This concern is decisive enough to reject the empirical claim as stated. Additional weaknesses — no RAG/fine-tuning baselines, untested MRS, no error bars — would remain even with a clean split, but the overlap alone invalidates the headline comparison. If a clean held-out re-run showed significant gains, the framework would still need baselines before the broader 'without retraining' claim could be accepted.","tokens_in":11698,"tokens_out":6050,"duration_ms":61451,"concrete_test":"Re-run Experiment 3 on the 61 records that were not used to build memory, using the memory store built from the 46 records, with the same GPT-3.5/GPT-4 and c=1 settings. Compare AICS, KPCS, and correct/incorrect counts against the Experiment 1 baseline on those same 61 records. If the improvement over baseline is not significant, the claimed memory injection effect is memorization of stored SME corrections. To make this possible, release the MedMCQA record IDs for the 46 memory-building records and the 107 evaluation records; alternatively, run a leave-one-memory-out protocol on all 107 records.","verdict_should_be":"REJECT","load_bearing_attack":"Section 3.1.3 states: 'We tested the memory injection framework on 107 records with built memory from the previous 46 records which had generated incorrect responses.' The paper nowhere states that these 107 records are disjoint from the 46. The published numbers determine the overlap: if the baseline for Figure 5 has 46 incorrect and 61 correct responses among the 107, then converting 31 of the 46 incorrect responses into correct ones yields exactly a 50.8% increase in correct responses (31/61) and a 67.4% reduction in errors (31/46). Section 4 reports exactly these percentages, and Experiment 4 then uses the 15 remaining incorrect responses from Experiment 3 (46 - 31 = 15). So the 46 memory-building records are included in the 107-record evaluation set. Because the memory for those records contains SME corrections derived from ground-truth explanations (Table 3), and because MRS was configured with c=1 (similarity-only, Section 1.2), cosine retrieval will surface the stored target content for those queries. The AICS/KPCS and correctness gains are therefore explained by looking up the answer for a large fraction of the test set rather than by any generalized memory refinement. The central claim that MARK enables continuous learning without retraining is not supported by an independent evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MARK, a memory-augmented framework that combines three agentic memory types (residual, user-question, and LLM-response refined memories) built by a Memory Builder Service and retrieved by a Memory Search Service using a weighted scoring formula. The authors claim that MARK enables LLMs to continuously learn without retraining. They evaluate on MedMCQA through four experiments: a baseline without memory, memory construction from 46 previously incorrect records, memory injection on 107 records, and a multi-turn multi-user experiment on 15 remaining incorrect records. Reported results include an AICS improvement from 0.18 to 0.36, KPCS from 0.12 to 0.32, a 50.8% increase in correct responses, and a 67.4% reduction in errors. However, the evaluation is not independent: the 107-record test set appears to include the 46 records used to build the memory, so the reported gains are likely explained by retrieval of stored SME corrections rather than by generalized refinement. The manuscript provides no code, no data, and no statistical uncertainty quantification.","tokens_in":12029,"tokens_out":5557,"duration_ms":53464,"significance":"If the empirical claims were valid, MARK would offer a practical, fine-tuning-free approach to domain adaptation for LLM-based chatbots, with plausible applications in enterprise settings. The decomposition of memory into three specialist agent types and the trust/persistence retention policy are design elements worth studying. However, the central claim is currently unsupported: the evaluation is circular, the sample sizes are very small, and key evaluation parameters are tuned on the same data used to report improvements. The paper does not establish that memory injection improves accuracy beyond memorization, nor does it clearly delineate its novelty relative to existing memory-augmented LLM work such as MemGPT and A-MEM. The reported effect sizes are therefore not credible as evidence of continuous learning.","major_comments":[{"comment":"The Experiment 3 evaluation is circular. The paper states that memory was built from the previous 46 records that generated incorrect responses and then tests on 107 records, but it never states that the 107 records are disjoint from those 46. The published numbers indicate they are not disjoint: if 31 of the 46 baseline incorrect responses are flipped to correct, the correct count increases by 31/61 = 50.8% and the error count decreases by 31/46 = 67.4%, which are exactly the percentages reported in Section 4. Experiment 4 then uses the 15 remaining incorrect responses from Experiment 3 (46 − 31 = 15). Because MRS is configured with c = 1 (similarity-only) and the memory store is populated with SME corrections derived from ground-truth explanations, the reported gains are most plausibly the result of retrieving stored answers for queries already used in memory construction. This invalidates the central claim that MARK enables continuous learning without retraining.","section":"3.1.3, Figures 4–5, Section 4"},{"comment":"The AICS weighting parameter α and the token-count filter are selected from the same baseline data on which the main improvements are later reported. The text states that α = 0.1 is chosen because it maximizes the separation between correct and incorrect responses in Figure 2, and that the 150–220 token region from Figure 3 is used to refine the dataset. Tuning evaluation parameters on the same data used to demonstrate improvement is a form of overfitting; it biases the reported metrics and inflates the apparent effectiveness of the framework.","section":"2.2.3, 3.1.1, Figures 2 and 3"},{"comment":"The memory store is populated with corrections derived from MedMCQA ground-truth explanations, and the SMEs are described as 'the individuals who came up with MARK.' Because the same queries that generated the incorrect responses are reused in the test set, the retrieval process for a large fraction of the test records is effectively a lookup of ground-truth-derived memory content. The reported AICS/KPCS gains are therefore a measure of nearest-neighbor retrieval of memorized examples, not a measure of learned refinement that generalizes to unseen questions.","section":"Table 3, Section 1.2"},{"comment":"The sample sizes are very small (46 memory-building conversations, 107 test records, 15 multi-turn sessions), and the paper reports no confidence intervals, standard errors, or statistical tests. The headline improvements of 100% (AICS) and 166.7% (KPCS) are point estimates computed on overlapping data, so their reliability cannot be assessed. This alone would prevent any strong conclusion about the framework's effectiveness even if the evaluation were properly disjoint.","section":"2.1, 3.1, Table 6"}],"minor_comments":[{"comment":"In the MRS formula, the symbols RC, Rec, SS, and FS are used before their definitions are given in the table; please define all symbols in the equation caption or immediately before the formula.","section":"1.2"},{"comment":"The phrase 'refining knowledge overtime' should be 'refining knowledge over time'; there are also minor grammatical errors such as 'discarding outdated information and refining knowledge overtime.'","section":"3.1.3"},{"comment":"The trust-score equation uses α with a different meaning than the AICS α in Section 2.2.3. The two α parameters should be given distinct names or explicitly distinguished in the text to avoid confusion.","section":"Appendix 1"},{"comment":"The conclusion states the '50.8% increase in correct responses and 67.4% reduction in errors' as an established result without acknowledging that these numbers depend on an evaluation set that may include the memory-building records; a caveat is needed until an independent evaluation is performed.","section":"Section 4"},{"comment":"Several references are incomplete (e.g., [12] lacks a venue, [15] has a truncated title 'LLM-Based'). Please complete the bibliography entries for the listed arXiv papers.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript resembles an industry technical report rather than a rigorous research paper. The absence of code, data, and any attempt at statistical validation makes it difficult to accept even after extensive revision. The core evaluation is structurally circular: the reported improvements are exactly what one would obtain by memorizing the 46 records used to build memory. This is not a local flaw but a failure of the paper's central demonstration, and it cannot be remedied without new experiments on a truly disjoint test set. The paper may be more appropriate for a workshop or industry venue after substantial methodological revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things up front. The MARK architecture is a reasonable incremental design: three memory agents plus a weighted scoring formula. That part is fine. But the headline result — that memory injection doubles AICS and cuts errors by two-thirds — is probably an artifact. Experiment 3 evaluates on 107 records using memory built from 46 prior records, and the paper never says those 107 are disjoint from the 46. The reported percentages line up exactly with what you'd get if 31 of the 46 incorrect records were flipped to correct: +50.8% correct (31/61) and −67.4% errors (31/46). That's too neat to be coincidence.\n\nWhat's actually new: the three-way decomposition (Residual, User Question, LLM Response) and the MRS formula with recall count, recency, similarity, and a feedback term. That extends A-MEM and MemGPT in a small but legitimate way. The appendix prompts are concrete and reproducible, which is more than most such papers provide.\n\nThe soft spots are real. The MRS scoring is never tested: for the experiments they set a=b=d=0, c=1, so it's pure cosine similarity. Alpha and the token thresholds are chosen from the baseline data, which is fitting, not prediction. There are no baselines against plain RAG, fine-tuning, or an existing memory system. Sample sizes are tiny (46, 107, 15) and there are no confidence intervals. Most importantly, the memory store contains SME corrections derived from MedMCQA ground-truth explanations. With similarity-only retrieval, the system is just looking up the answer it already stored. The demonstration shows retrieval of stored corrections, not continuous learning or generalized refinement.\n\nThe framework description is coherent and the authors acknowledge some risks (trust/persistence scores), but the central empirical claim is unsupported. I see no internal contradiction, so the thinking is serious; the evaluation is not.\n\nMy recommendation: this deserves a referee's time only if the authors can rerun Experiment 3 on a truly disjoint test set and compare against a RAG baseline. On the current evidence I'd lean reject. If I were the editor, I'd ask the authors for that experiment before sending it out — or send it out with the overlap question as the first thing the referee checks.","headline":"MARK's architecture is a reasonable incremental design, but its core result is likely a retrieval artifact from overlapping test and memory-building records.","tokens_in":12533,"tokens_out":3602,"would_cite":false,"duration_ms":33241,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that injecting structured refined memory into an LLM's context lets it learn from expert corrections without retraining, with reported accuracy gains on medical questions.","keywords":["LLM memory","agentic memory","knowledge refinement","memory injection","Medical question answering","hallucination reduction","fine-tuning alternative","Memory Relevance Scoring"],"falsifier":"Compute the overlap between the 46 memory-building records and the 107 test records in Experiment 3; if any test question or its ground-truth explanation was used to construct the injected memory, rerun on a disjoint holdout and check whether AICS and KPCS gains persist.","tokens_in":11509,"feed_emoji":"🧠","tokens_out":5442,"duration_ms":49730,"temperature":0.7,"pith_summary":"The paper proposes MARK, a memory layer that lets a large language model chatbot absorb subject-matter-expert corrections and evolving domain knowledge without retraining or fine-tuning. Three specialized agents extract compact 'refined memory' from conversations—residual domain insights, user-provided terminology, and response patterns that users accepted—and a retrieval service injects the most relevant memories into the prompt at generation time. In experiments on medical multiple-choice questions, injecting this memory raised the average information capture score from 0.18 to 0.36 and the key-point coverage score from 0.12 to 0.32, while correct responses increased by 50.8% and errors fell by 67.4%. The point of the framework is to make high-stakes chatbots accurate and current through structured memory rather than expensive periodic retraining.","feed_headline":"Memory injection lifts medical QA accuracy by 50.8%","feed_subtitle":"Structured refined memory from expert corrections lifts AICS from 0.18 to 0.36 and cuts errors by 67.4%.","key_machinery":"The load-bearing object is the Memory Relevance Scoring (MRS) function, $MRS = a\\,RC + b/(Rec+\\epsilon) + c\\,SS + d\\,FS$, with $a+b+c+d=1$. RC counts how often a memory has been retrieved, Rec is the age of the memory in days, SS is the semantic similarity between memory and query, and FS is a feedback score of $+1$, $0$, or $-1$ for correct, unrated, or incorrect feedback. This score selects which refined memories are injected, and each agent type contributes its highest-scoring memory to the final prompt. For the experiments the authors set $a=b=d=0$ and $c=1$, so in this paper the mechanism is driven solely by query–memory similarity; the other weights are presented as tunable knobs for recency-, popularity-, or feedback-sensitive deployments.","core_discovery":"On its own terms, MARK claims that a language model can continuously learn by having its context augmented with structured refined memory built from past conversations. A Memory Builder Service watches dialogues between users and the assistant, extracting residual memory (implied domain logic), user-question memory (terms and abbreviations), and LLM-response memory (what the user accepted and why). A Memory Search Service then retrieves the top memories by similarity to the new query, reranks them with a weighted score that combines recall count, recency, similarity, and feedback, and injects the best memory of each type into the generation prompt. Using a medical multiple-choice dataset, the authors report that this injection improved factual alignment substantially while keeping information coverage constant, and that most expert corrections were captured within two to three conversational turns. The intended consequence is that domain knowledge can be refined by experts and users across sessions without model retraining.","pith_inferences":["The reported 100% AICS gain may partly reflect retrieval of the exact ground-truth explanations used to build the memory, since the SME corrections were based on the dataset's explanations; an independent holdout set is needed to separate memorization from generalization.","MARK could be paired with a frozen retrieval-augmented-generation pipeline, with refined memory acting as an expert-validated overlay that corrects and updates the static corpus.","The trust-score and persistence-score mechanism in the risk section suggests a testable experiment: deliberately inject contradictory memories and measure how quickly feedback demotes them.","An ablation varying the MRS weights would reveal whether the default weight combination is necessary or whether similarity alone explains the reported improvement."],"forward_implications":["An SME correction made in one session can persist across later sessions and across users, so the same error need not be re-corrected.","Domain chatbots can stay current with evolving terminology and regulations by accumulating roughly ten structured memory units per correction conversation, without retraining.","Weighting MRS toward feedback and recency would let high-stakes domains prioritize expert-validated memories over older ones.","Because memory is stored as text plus metadata, enterprises can apply data-governance and leakage-prevention controls per memory.","If the gains replicate on disjoint test sets, memory injection offers a cheaper alternative to fine-tuning for domain adaptation."],"supporting_citations":[{"why":"Supplies the Society of Mind idea of breaking intelligence into specialized agents, which motivates the three-memory-agent design.","marker":"[5]"},{"why":"Establishes that fine-tuning can adapt LLMs but is computationally expensive, the cost MARK aims to avoid.","marker":"[7]"},{"why":"Documents failure points in retrieval-augmented generation, motivating the move to memory that can be refined in place.","marker":"[8]"},{"why":"Provides the survey of LLM hallucination factors that motivates ground-truth-anchored memory.","marker":"[10]"},{"why":"Supports the agentic memory building and retrieval approach for LLM agents that MARK extends.","marker":"[11]"},{"why":"Shows that adding memory to the context improves LLM accuracy, the premise behind injection.","marker":"[14]"},{"why":"Supplies the relevance, recall frequency, and elapsed time factors that MRS adapts into its score.","marker":"[15]"},{"why":"Supplies the MedMCQA dataset and expert-verified medical ground truth used for building memory and evaluation.","marker":"[16]"},{"why":"Establishes the accuracy and pointwise-score benchmarks on MedMCQA that the paper replaces with its coverage metrics.","marker":"[17]"}],"fun_headline_variants":["Memory agents boost medical QA accuracy 50.8%","No retraining: memory injection cuts LLM errors 67%","MARK: LLMs learn continuously via structured memory","Residual memory lifts QA accuracy 50.8%","LLM memory refinement lifts accuracy without fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains rest on the assumption that the 107 test questions do not overlap with the 46 corrected conversations used to build the injected memory, so that improvement reflects generalization rather than recall of stored answers.","fun_headline_variants_meta":{"raw":{"variants":["Memory agents boost medical QA accuracy 50.8%","No retraining: memory injection cuts LLM errors 67%","MARK: LLMs learn continuously via structured memory","Residual memory lifts QA accuracy 50.8%","LLM memory refinement lifts accuracy without fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000584,"raw_usage":{"total_tokens":2776,"prompt_tokens":1003,"completion_tokens":1773,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":1694}},"tokens_in":619,"tokens_out":1773,"duration_ms":12175,"temperature":1.0,"reasoning_tokens":1694,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:10:35.897719+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the overlap between the 46 memory-building records and the 107 test records in Experiment 3; if any test question or its ground-truth explanation was used to construct the injected memory, rerun on a disjoint holdout and check whether AICS and KPCS gains persist.","supporting_citations":[{"cited_title":"Minsky, Society of Mind, Simon & Schuster, 1986","cited_arxiv_id":null,"evidence_quote":"Supplies the Society of Mind idea of breaking intelligence into specialized agents, which motivates the three-memory-agent design."},{"cited_title":"MedMCQA: A Large-scale Multi- Subject Multi-Choice Dataset for Medical domain Question Answering,","cited_arxiv_id":null,"evidence_quote":"Supplies the MedMCQA dataset and expert-verified medical ground truth used for building memory and evaluation."}],"review_version":1}