{"id":"609e3f06-56c0-459c-a209-4ffdf1abc538","arxiv_id":"2508.04289","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An external store of problem-solution prompt pairs is retrieved and injected into LLM queries, and toy experiments claim this improves factual checking and generalizes across made-up software names.","lead":"Large language models get a memory of reusable problem-solution steps, called methods, that are retrieved and injected into prompts on new queries. The paper reports that this improves how often the model checks whether a fake software product exists before writing setup instructions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Experiment's success metric is keyed to the injected method's software name, so the reported generalization gain may be a lexical artifact.","rationale":"The reader's verdict is REJECT, and my stress-test supports that verdict; I therefore recommend no change. I partially agree with the reader's weakest_assumption: the relevance-transfer assumption in Eq. 20 is indeed unproven, but the more load-bearing flaw is even more concrete—the evaluation metric itself is designed around the injected method's specific software name. The system is never tested on whether the retrieved method produces the correct answer for the target software; it is tested on whether it echoes the instruction. This makes the reported improvement tautological and leaves the central claims of generalization and continuous improvement unsupported. A corrected reference sentence or binary correctness label would settle the issue.","tokens_in":12628,"tokens_out":3897,"duration_ms":45546,"concrete_test":"Recompute the §IV-C comparison with the reference sentence changed to 'Verify whether HongHanKey is a real and identifiable piece of software' (or, better, use a binary ground-truth label: does the response state that HongHanKey cannot be found?). If method1's mean cosine similarity is no longer above the NoMethod baseline, the reported generalization effect is an artifact of the SuHongKey-named reference, and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's only quantitative support for the central claim is in §IV-C. There, cs2 injects method1 = 'check whether SuHongKey exists'; cs3 asks about HongHanKey; and the evaluation compares outputs against compareResult = 'Verify whether SuHongKey is a real and identifiable piece of software.' Thus a response that repeats a SuHongKey existence check—exactly what method1 naively tells the model to do—is scored high, while a response correctly checking HongHanKey is scored low. The 0.7791 vs 0.4693 gap therefore measures adherence to the injected instruction, not factual verification or generalization to a new software name. The §III-D 'proof' (Eq. 20) only asserts that relevance exceeding a threshold makes a method viable; it does not establish that relevance transfers correctness. In §IV-D, method2 is manually supplied as 'check whether the target software exists' and then compared to a reference that matches that instruction, so it also does not demonstrate user-driven refinement of an extracted method. Without a metric tied to the actual correct answer (HongHanKey does not exist), the central claim is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a method-based reasoning layer for LLMs, in which reusable procedures are extracted from training content, LLM outputs, and user interactions, represented as problem-solution pairs, stored in a tree, retrieved by relevance, and selected by LLM/user ranking. The authors formalize retrieval with a utility function and a threshold-based generalization claim, then evaluate the approach with GPT-4o on two nonexistent software names (SuHongKey and HongHanKey), reporting cosine-similarity gains for method reuse and for a newly learned general method.","tokens_in":12889,"tokens_out":6546,"duration_ms":79798,"significance":"The idea of an explicit, persistent procedural memory that is independent of surface text is timely and could be a useful complement to CoT, RAG, and RLHF, particularly for verification-before-generation tasks. The paper is clearly structured and sketches a concrete pipeline. However, the current manuscript provides no valid quantitative evidence for the central claims: the reference sentences are aligned with the injected methods, the proof of generalization is a restatement of an assumption, and no error bars, baselines, or implementation details are given. The contribution is therefore more of a position/architecture sketch than a validated system.","major_comments":[{"comment":"The 'Brief Proof' does not prove generalization. It asserts that Relevance(p_new, p_i) >= theta makes method_i a viable candidate, but no argument connects relevance to the correctness or effectiveness of s_i for p_new; the 'solves' relation in Eq. (2) is assumed, not established. The load-bearing premise - semantic similarity transfers solution validity - is exactly what needs proof or empirical support. As written, the section restates the method's selection rule rather than justifying it.","section":"III-D, Eq. (20)"},{"comment":"The success metric is keyed to the injected instruction. The cs3 prompt asks about HongHanKey, but method1 instructs checking SuHongKey and compareResult is 'Verify whether SuHongKey is a real...'. A response that recites the SuHongKey check is scored high, while a correct response checking HongHanKey is scored low. The 0.7791 vs 0.4693 gap therefore measures adherence to the supplied method, not factual verification or generalization. The absence of error bars and significance tests further weakens the claim.","section":"IV-C, Figure 2"},{"comment":"compareResult2 is semantically almost identical to method2's instruction ('Please check whether the target software exists or not...'). Comparing outputs against that sentence is circular. Moreover, method2 is manually supplied in ics2; there is no extraction or user-driven ranking/refinement step. Thus the experiment cannot support the abstract's claim that 'newly learned methods can outperform earlier ones through user-driven refinement.'","section":"IV-D, Figure 3"},{"comment":"The implemented retrieval does not follow the formal model. Eqs. (17)-(18) use Effectiveness(s_i), but the experiments never operationalize this term, and Eq. (14) threshold tau / Eq. (20) theta are never set or ablated. Retrieval is described qualitatively as the MethodManager 'evaluates similarity' and then delegates to the LLM. Without implementation details, code, or data, the empirical results cannot be reproduced or compared to baselines.","section":"III-C and IV-C"}],"minor_comments":[{"comment":"The figures report point estimates over 20 trials with no error bars, variance, or statistical tests; any retained quantitative claim should include uncertainty and significance testing.","section":"Figures 2 and 3"},{"comment":"The thresholds tau and theta are described as tunable, but no tuning procedure or sensitivity analysis is given. A small ablation would help the reader understand how robust the reported behavior is.","section":"Eqs. (14) and (20)"},{"comment":"There are template/formatting artifacts (e.g., 'JOURNAL OF LATEX CLASS FILES' header, 'PLACE PHOTO HERE' in the author biography) and inconsistent notation (e.g., 'method_i in M_candidate' and the use of p_i/s_i) that should be cleaned.","section":"Throughout"},{"comment":"Algorithm 1 is not fully connected to the implemented MethodManager. Steps such as 'rank candidates' and 'user selects or ranks the best output' are not operationalized in the experiments, making it hard to see how the formal procedure maps to the reported results.","section":"Algorithm 1 and IV-C"}],"recommendation":"reject","confidential_remarks":"The experimental design appears to be constructed so that the comparison sentences overlap with the injected methods; this is a substantive flaw, not merely a presentation issue. The paper would need a new evaluation task, a correct reference for the actual target software, proper baselines, and statistical reporting to be considered. The formal proof is also not a proof, so a substantially reworked submission would be required."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi,\n\nShort take: this is a coherent position paper for an external 'method library' that stores problem-solution pairs and injects them into prompts. The architecture is reasonable and clearly described, and the related work is honestly surveyed. What it is not is a validated system. The two quantitative experiments are keyed to the injected instructions, so they demonstrate instruction-following, not generalization or continual learning.\n\nThe concrete problems: Section III-D's 'Brief Proof' is just a restatement of the relevance threshold (Eq. 20); it does not show that relevance transfers correctness. In IV-C, the reference sentence is 'Verify whether SuHongKey is a real and identifiable piece of software,' but the test prompt is about HongHanKey. method1 explicitly tells the model to check SuHongKey, so a response that mechanically repeats that check scores higher against the reference than a response that actually checks HongHanKey. The 0.7791 vs 0.4693 gap measures prompt adherence, the very thing the injected method is designed to produce. Section IV-D has the same problem: method2 is manually supplied as 'check whether the target software exists,' and compareResult2 says 'No official or widely recognized software named HongHanKey could be found.' The metric and the method are near-duplicates, so the comparison is tautological.\n\nThere's no code, no error bars, no statistical tests, and no baseline against RAG or a simple 'verify existence' instruction. The 'automatic extraction' is not shown; in both experiments the method is explicitly typed by the user.\n\nWhat's genuinely useful here is the framing: storing a procedure as a problem-solution pair, with dual ranking by user and LLM, and part-wise refinement. That's a sensible incremental extension of retrieval-augmented prompting. But the paper's own evaluation can't separate this architecture from the trivial effect of appending a relevant instruction to the prompt.\n\nWho is this for? Someone building memory-augmented prompting systems might find the vocabulary useful, and the paper could serve as a workshop discussion piece about evaluation design. But as an arXiv submission, it's not ready for a serious referee in its current form. The right next step is a proper experiment: same software name across sessions, a metric that checks the actual target, comparisons against RAG and a no-method baseline, and enough trials for a significance test.\n\nI'd reject it now, but I'd encourage the author to resubmit with a real evaluation.","headline":"A coherent architecture for procedural memory, but the experiments measure adherence to the injected instruction, not generalization.","tokens_in":13359,"tokens_out":3340,"would_cite":false,"duration_ms":35855,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that explicit, reusable problem-solution pairs stored outside the LLM—retrieved by semantic relevance and ranked by feedback—let language models reason more consistently, generalize to new problems, and improve over time.","keywords":["large language models","method extraction","procedural memory","problem-solution pairs","retrieval-augmented reasoning","continual learning","chain-of-thought","factual verification"],"falsifier":"Take the cs1/cs3 setup but store a method whose solution is correct for its own problem yet wrong for a semantically similar new problem, such as a stored existence-check procedure retrieved for a prompt about real, widely known software; if the relevance condition of Eq. (20) passes and the system applies the inappropriate procedure, the generalization claim fails. A second concrete check is to measure how often the LLM, when given a retrieved method, rejects or revises it instead of applying it; a high rejection rate would indicate that semantic relevance alone does not transfer solutions.","tokens_in":12501,"feed_emoji":"🧠","tokens_out":5835,"duration_ms":71041,"temperature":0.7,"pith_summary":"Large language models generate text by predicting the next token, so they can invent steps when a prompt looks procedurally rich but the premise is false. This paper proposes adding an external procedural memory: a library of methods, each stored as a problem paired with a solution procedure, extracted automatically from training content, generated responses, and user interactions. When a new query arrives, the system retrieves the most relevant stored methods and lets the LLM apply them; user and LLM rankings decide which method wins. The reported experiments use fake software names and show that a learned method raises the alignment of responses with the intended existence check from 0.4693 to 0.7791, and that a later, more general method outperforms an earlier one, 0.8426 versus 0.4587. A sympathetic reader cares because the claim is a concrete path to continual, auditable reasoning improvement without retraining the model.","feed_headline":"External method store lifts LLM fact-check score from 0.47 to 0.78","feed_subtitle":"A reusable problem-solution library lets a language model verify facts before answering and improves as users refine it.","key_machinery":"The central object is the method, an explicit problem-solution pair, organized into a method storage tree whose nodes are problems and whose edges are semantic generalization or specialization relationships. The load-bearing mechanism is retrieval by relevance: a stored method is treated as applicable to a new query when semantic similarity between the problems passes a threshold, and the ranking function combines user feedback with LLM judgment. This machinery turns reasoning into a selection-and-application process: retrieve candidate methods, rank them, apply the best to guide generation, then store the new problem-solution association for future use.","core_discovery":"The paper's central claim is that LLM reasoning can be improved by decoupling problem understanding from solution execution. Each method is a logical pair, $\\mathrm{Method} = \\{\\mathrm{problem}, \\mathrm{solution}\\}$, stored in a method storage tree and selected for a new problem $p_{\\mathrm{new}}$ when $\\mathrm{Relevance}(p_{\\mathrm{new}}, p_i) \\geq \\theta$. The system ranks candidates by a logical utility $U(m_i, p_j) = \\mathrm{Relevance}(p_i, p_j)\\cdot \\mathrm{Effectiveness}(s_i)$, combining external user ranking (RLHF-style feedback) with internal LLM selection. The paper argues that because the solution component is detached from surface-level phrasing, a procedure learned for one proble","pith_inferences":["Editorial inference: the same repository idea could be pushed further by storing methods as executable code or formal preconditions, so that the applicability of a method is checked by the method's own preconditions rather than by semantic similarity alone.","Editorial inference: a testable extension is to log every retrieval where the LLM rejects or revises the supplied method; these rejection cases would define an empirical failure surface for the relevance threshold and could seed better problem representations.","Editorial inference: when a later user correction contradicts an earlier stored method, the ranking mechanism implies a version-like conflict resolution, but the paper does not address contradictory methods explicitly.","Editorial inference: the reported cosine-similarity metric rewards wording alignment with a reference sentence; measuring factual accuracy directly, such as whether the model actually checked existence before continuing, would make the generalization claim more robust."],"forward_implications":["If the central claim holds, LLM applications can accumulate a shared library of verified procedures that persists across sessions and users, so correctness no longer depends on a single prompt or on the model's memorized patterns.","A stored method can act as a guardrail: before answering a procedural question, the system can apply a learned precondition such as checking whether the target software exists, preventing fabricated setup steps.","User feedback becomes a continuous-improvement mechanism: a generalized method learned from a later interaction can outrank and replace an earlier, narrower method, as shown by method2 beating method1.","Because methods are stored externally and can be inspected, a failure can be traced to a specific procedure rather than to opaque model weights.","The method repository sits on top of the LLM rather than inside it, so the approach can be combined with existing prompting, retrieval, and tool-use techniques without retraining the underlying model."],"supporting_citations":[{"why":"Supplies the baseline chain-of-thought prompting technique whose static, manually designed nature the method-based model is positioned to overcome.","marker":"[8]"},{"why":"ReAct, the reasoning-and-acting baseline, is contrasted as relying on crafted prompts rather than reusable stored methods.","marker":"[9]"},{"why":"Supplies the RLHF-style user preference feedback mechanism that the method-ranking and refinement loop is built on.","marker":"[10]"},{"why":"RAG is the retrieval baseline; the paper distinguishes retrieving external documents from retrieving abstract solution procedures.","marker":"[11]"},{"why":"Toolformer provides the precedent of LLMs calling external tools, supporting the paper's design of external executable methods.","marker":"[14]"},{"why":"Self-improvement work is the continual-learning baseline; the paper argues it lacks a persistent, structured procedural memory.","marker":"[18]"}],"fun_headline_variants":["LLMs get a method library to reason beyond next-token prediction","Reusable problem-solution pairs lift LLM fact-check to 0.78","LLMs store problem-solving methods for continual self-improvement","Method-based reasoning: LLMs extract, reuse, and improve procedures","From next-token stats to explicit reusable methods for LLM reasoning"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The argument collapses if semantic relevance between the stored problem and the new problem—judged by the LLM or by embedding similarity—is not enough to guarantee that the stored solution procedure actually applies.","fun_headline_variants_meta":{"raw":{"variants":["LLMs get a method library to reason beyond next-token prediction","Reusable problem-solution pairs lift LLM fact-check to 0.78","LLMs store problem-solving methods for continual self-improvement","Method-based reasoning: LLMs extract, reuse, and improve procedures","From next-token stats to explicit reusable methods for LLM reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000761,"raw_usage":{"total_tokens":3195,"prompt_tokens":705,"completion_tokens":2490,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":2398}},"tokens_in":449,"tokens_out":2490,"duration_ms":21337,"temperature":1.0,"reasoning_tokens":2398,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:43:08.463785+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the cs1/cs3 setup but store a method whose solution is correct for its own problem yet wrong for a semantically similar new problem, such as a stored existence-check procedure retrieved for a prompt about real, widely known software; if the relevance condition of Eq. (20) passes and the system applies the inappropriate procedure, the generalization claim fails. A second concrete check is to measure how often the LLM, when given a retrieved method, rejects or revises it instead of applying it; a high rejection rate would indicate that semantic relevance alone does not transfer solutions.","supporting_citations":[{"cited_title":"Training language models to follow instructions with human feedback,","cited_arxiv_id":null,"evidence_quote":"Supplies the RLHF-style user preference feedback mechanism that the method-ranking and refinement loop is built on."},{"cited_title":"Retrieval-augmented generation for knowledge- intensive nlp tasks,","cited_arxiv_id":null,"evidence_quote":"RAG is the retrieval baseline; the paper distinguishes retrieving external documents from retrieving abstract solution procedures."},{"cited_title":"Toolformer: Language models can teach themselves to use tools,","cited_arxiv_id":null,"evidence_quote":"Toolformer provides the precedent of LLMs calling external tools, supporting the paper's design of external executable methods."}],"review_version":1}