{"id":"8494806a-fd7a-49c6-b092-941dfd449c06","arxiv_id":"2501.06590","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A self-updating library of decomposed chemistry sub-tasks improves LLM accuracy on SciBench chemistry problems, with reported gains of up to 46 percentage points over direct reasoning.","lead":"ChemAgent gives large language models a searchable library of chemistry sub-problems and solutions, built from a small development set and updated as new problems are solved. On four college chemistry datasets it reports large accuracy gains over standard prompting, but the evaluation protocol raises questions about how much of the gain comes from using test-set information.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline gains in Tables 1 and 4/5 may be inflated by updating the memory from test-set problems during inference, so the comparison against static baselines is not yet apples-to-apples.","rationale":"The paper is honest about its own failure modes: Section 2.11 explicitly documents an adiabatic/isothermal retrieval failure, and the ablations suggest memory is helpful on average. The retrieval-similarity concern raised by the reader is real but not the single most load-bearing issue: imperfect retrieval can still leave positive average gains, and the paper itself provides evidence that higher-similarity retrievals correlate with correctness. The more serious issue is that the evaluation protocol may let the method see the test set through its own running memory updates. This directly affects every headline number, not just error cases. The reader flagged this in the rationale as a 'serious transparency issue', but did not make it the weakest assumption, hence partial agreement. The proposed test is straightforward and should be feasible with the released code: toggle the memory-update flag and rerun. If the gains persist in a strict static setting, the central claim survives; if not, the paper should be revised to distinguish one-pass generalization from transductive self-evolution. Numerical inconsistencies in the reported relative gains (e.g., '2.93 times' applied to StructChem) reinforce the need for a clean, protocol-matched rerun.","tokens_in":26982,"tokens_out":6125,"duration_ms":62177,"concrete_test":"Re-run the GPT-4 configuration in Table 1 with a single code change: build Mp/Me only from Dd via Algorithm 1 and disable all Me/Mp insertions during Dt inference, keeping Evaluation & Refinement active. If ChemAgent's CHEMMC accuracy falls from 74.36 toward the 58.97 'w/o Memory' row and the four-dataset average gain over StructChem shrinks materially, the headline improvement is substantially attributable to test-set memory updates. Additionally, or alternatively, split Dt into a disjoint warm-up set (allowed to update memory) and an evaluation set, and report accuracy on the evaluation set only; if the gain over static baselines disappears, the self-evolution claim needs to be reframed as transductive rather than generalizable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.5 states that during library-enhanced reasoning 'the memory is continuously updated with newly solved sub-tasks and their solutions during runtime,' with Me = Me ∪ {(Cj,Tj,Oj)} and Mp updated similarly, and Figure 2 labels the pipeline as 'test set' with an '(Updated) Library.' If this dynamic update is active in Table 1, each earlier test problem can seed memory for later test problems. Section 2.9's exclusion only prevents a problem from being re-solved with memory derived from that same problem in a previous iteration; it does not prevent cross-problem leakage within the test set. With only 34-107 test examples per dataset, a handful of cached solutions can move per-dataset accuracy by several percentage points. The baselines (Direct reasoning, Python, StructChem) never receive this test-set exposure, so the reported 46% CHEMMC gain and the 57.16 vs. 47.66 average may reflect transduction rather than generalizable chemical reasoning. The paper should separately report a static-library run and a self-evolving run; currently it is ambiguous which setting produced Tables 1, 4, and 5.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ChemAgent is an agentic framework for chemical reasoning that combines a structured library with three memory types (planning, execution, and knowledge) with LLM-based task decomposition, retrieval, and an evaluation-and-refinement module. The library is initially constructed from development-set problems by decomposing them into sub-tasks with conditions, questions, and solutions; during inference, new sub-tasks are matched to stored memory by Llama3-embedding cosine similarity, and the library is updated with newly solved sub-tasks and their solutions. On four SciBench chemistry datasets, the authors report that GPT-4-based ChemAgent reaches 57.16% average accuracy versus 47.66% for StructChem and 19.48% for direct reasoning, with a claimed gain of up to 46% on CHEMMC, and they report ablations of memory components, a memory-quality analysis, and a self-evolution experiment.","tokens_in":27225,"tokens_out":8499,"duration_ms":78966,"significance":"If the reported evaluation is valid, the contribution is significant: ChemAgent provides a concrete, fully prompted instantiation of memory-augmented decomposition for a domain where LLMs are known to make cascading errors, and the paper ships open-source code and detailed prompts in Appendix G. The ablations of memory components and memory quality, the cost analysis, and the self-evolution curve in Figure 5 are informative and go beyond a single accuracy number. However, the significance rests on the main comparison being apples-to-apples with static baselines, and that is not currently established because the runtime memory-update protocol exposes the test set to information that baselines never receive. The central claims of large gains over StructChem/direct reasoning and of self-evolution therefore need a clean static-versus-dynamic comparison before they can be taken at face value.","major_comments":[{"comment":"The main comparison is confounded by transductive test-set exposure. Section 2.5 states that during library-enhanced reasoning \"the memory is continuously updated with newly solved sub-tasks and their solutions during runtime,\" with Me = Me ∪ {(Cj,Tj,Oj)} and Mp updated similarly, and Figure 2 labels the pipeline as operating on the \"test set\" with an \"(Updated) Library.\" This means that, under the protocol as written, an earlier test problem can seed memory used by later test problems. Section 2.9's exclusion rule only prevents a problem from being re-solved with memory derived from that same problem in a previous iteration; it does not prevent cross-problem information flow within the test set. With test sets of only 34-107 examples, a small number of cached solutions can move per-dataset accuracy by several percentage points. The baselines (Direct reasoning, Python, StructChem) never receive this exposure. Please report separately (i) a static-library run in which no test examples are written to memory and (ii) a self-evolving run with memory updates, and state explicitly which setting produced each row of Tables 1, 4, and 5.","section":"§2.5, Fig. 2, Tables 1, 4, 5"},{"comment":"The self-evolution experiment does not define how a solution is judged \"correct\" before being added to the library. Section 2.9 says that \"all related responses and knowledge from that process are added to the library if the solution is correct,\" but no correctness-checking mechanism is described. If \"correct\" is determined by comparing against the test-set ground-truth answer, then the self-evolution curve in Figure 5 is fitted to the test set and its convergence to 44.89% is not evidence of generalizable self-improvement. Please specify the correctness criterion, and if it uses test labels, re-run with a label-free criterion or report both label-based and label-free versions.","section":"§2.9, Fig. 5"},{"comment":"The main GPT-4 results lack error bars or any indication of run-to-run variance. In Table 1, differences such as ATKINS 61.18 vs. 59.81 (StructChem) and QUAN 44.12 vs. 41.18 (StructChem) are on the order of one or two test examples on datasets of 107 and 34 items, respectively. Appendix B states that open-source-model runs were repeated at least three times, but no variance is reported for GPT-4, and Table 1 gives single numbers. Please report repeated runs, seeds, and confidence intervals for the main comparisons, or at minimum report per-seed scores. Without this, the headline 9.5-point average gap cannot be separated from sampling noise.","section":"Table 1, §2.7, §2.8"},{"comment":"The retrieval mechanism assumes that Llama3-embedding cosine similarity between sub-task texts tracks chemically meaningful problem structure, and the quality gate in Algorithm 1 assumes that LLM-assigned confidence correlates with factual correctness. Section 2.11 gives a concrete counterexample: a memory about entropy change during isothermal compression is retrieved for an adiabatic compression problem because the wording is similar, and the wrong strategy is used. The paper does not quantify how often such misleading retrievals occur or whether the confidence filter removes them. Because the library's benefit is the central claim, please report retrieval precision (for example, expert annotation of retrieved memories or a targeted error-case analysis) and the correlation between confidence scores and solution correctness.","section":"§2.5, §2.11, Algorithm 1"}],"minor_comments":[{"comment":"The caption says accuracy scores are computed using \"the approximation detailed in Section 4.3,\" but Section 4.3 does not exist; the main text gives only the relative tolerance of 0.01. Please state the full metric definition in the main text and correct the reference.","section":"Table 1 caption, §2.7"},{"comment":"The phrase \"performance gains of up to 46%\" is ambiguous: the underlying numbers are 28.21 vs. 74.36 on CHEMMC, which is an increase of 46 percentage points and a relative increase of roughly 163%. Please report the gain in consistent units.","section":"Abstract, §2.8"},{"comment":"Several hyperparameters are never given numerical values: the similarity threshold θ, the LLM-confidence threshold for discarding memory units, the maximum number of refinement operations, the accuracy tolerance, and the top-k for imagined problems. If these are set in the released code, please state their values or point to the exact configuration file in the main text.","section":"§2.5, Algorithm 1, §2.7"},{"comment":"The Table 5 title reads \"Llama 3.1-70b, Llama 3.1-70b and Qwen 2.5-72b,\" which appears to duplicate the Llama 3.1-70b entry; the intended list is likely Llama 3.1-7b, Llama 3.1-70b, and Qwen 2.5-72b.","section":"Appendix B, Table 5"},{"comment":"The sentence \"improves by 9.50% (47.66 vs. 57.16) over StructChem, which is a 2.93 times increase and by 37% (19.48 vs. 57.16) over direct reasoning\" mixes percentage-point gains with relative increases. Please rephrase to distinguish absolute percentage-point changes from multiplicative improvements.","section":"§2.8"}],"recommendation":"major_revision","confidential_remarks":"The paper is a candidate for acceptance after a corrected evaluation. The central obstacle is the test-set exposure in the main protocol and the undefined correctness criterion in the self-evolution experiment; if those cannot be resolved, the headline gains should be substantially discounted. I would not reject at this stage because the method, ablations, and released code are informative and the main technical idea is plausible. A static-library run and a clearly separated self-evolving run would settle the key question."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the method is a sensible integration of decomposition, retrieval, and self-refinement, and the ablations are more thorough than average. But the evaluation as reported doesn't let you separate the contribution of the library from the contribution of test-set exposure. Section 2.5 says the memory is 'continuously updated with newly solved sub-tasks and their solutions during runtime,' and the figure labels the pipeline 'test set' with an '(Updated) Library.' Baselines like StructChem and direct reasoning never get that dynamic memory. With only 34-107 test examples per dataset, a few cached solutions can move accuracy by several points. The paper needs a static-library run and a self-evolving run reported separately; right now it's ambiguous which one produced Tables 1, 4, and 5. This is the load-bearing issue.\n\nWhat's actually good: the three-memory design (plan, execution, knowledge) is a clean decomposition, and the idea of generating synthetic execution memory when retrieval comes up empty is worth borrowing. The ablations in Table 2 and the memory-quality analysis in Table 3 are directionally sensible. The error analysis in Section 2.11 is honest, and the prompts in Appendix G are detailed enough to reproduce. The code is public. That's real evidence the framework works as intended in a nonleaky setting, even if the magnitude is unproven.\n\nSmaller problems: the accuracy metric references a nonexistent Section 4.3; the abstract says 46% gain while the conclusion says 36%; and the '2.93 times increase' in Section 2.8 is numerically garbled. None of these change the method, but they make the paper harder to trust. The retrieval step relies on Llama3 embedding cosine similarity, and Section 2.11's own example (isothermal vs adiabatic compression) shows that similarity can pick the wrong strategy. That's a known limitation, not a hidden flaw. The quality gates in Algorithm 1 are self-referential—the LLM generates memory and scores it—but the final accuracy is measured against external answers, so the central claim is not circular.\n\nBottom line: this deserves a serious referee, but with a request for a clean separation between static and dynamic memory, and a pass to fix the numerical inconsistencies.","headline":"A useful agent framework for chemical reasoning, but the headline gains need re-measurement under a static library before they're trustworthy.","tokens_in":27800,"tokens_out":2429,"would_cite":false,"duration_ms":21309,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ChemAgent claims that a language model can solve multi-step chemistry problems more accurately when it builds and updates a structured library of previously solved sub-tasks.","keywords":["chemical reasoning","large language models","self-updating memory","task decomposition","retrieval-augmented reasoning","self-evolution","evaluation and refinement","college chemistry benchmarks"],"falsifier":"Give the agent two problems that read almost identically but differ in a condition that changes the governing formula, such as isothermal versus adiabatic compression; if the high-similarity memory is retrieved and its strategy is followed to the wrong answer even after the refinement step, the similarity proxy is the failure point. A systematic version would count how often accuracy drops when every test problem has a high-similarity distractor memory that shares vocabulary but requires a different formula.","tokens_in":26779,"feed_emoji":"🧪","tokens_out":8995,"duration_ms":127197,"temperature":0.7,"pith_summary":"ChemAgent claims that a large language model can reason through multi-step chemistry problems much more accurately if it keeps a structured library of previously decomposed sub-problems and their solutions, and updates that library as it works. The paper builds the library by splitting solved development problems into atomic sub-tasks, each paired with conditions and a sub-solution, and organizes them into three memory types: planning strategies, execution traces, and chemical knowledge. At test time, the model decomposes a new problem, retrieves similar stored sub-tasks by embedding similarity, refines its plan and answers, and writes newly solved sub-tasks back into the library. On four college-chemistry problem sets, the authors report average accuracy of 57.16% with the strongest tested model, versus 19.48% for direct reasoning and 47.66% for the prior structured-prompting baseline, with per-dataset gains up to 46 percentage points. The intended payoff is a framework that improves with experience rather than relying on fixed prompts or human-curated demonstrations.","feed_headline":"Self-updating memory lifts LLM chemistry accuracy by 46%","feed_subtitle":"Past sub-problems become reusable memory, lifting accuracy above direct reasoning and structured prompting.","key_machinery":"The load-bearing mechanism is a dynamic library with three memory types: planning memory stores high-level strategies; execution memory stores structured units of condition, sub-task, and sub-solution; and knowledge memory stores formulas and principles generated for the current problem. Sub-tasks are the atomic blocks: a new problem is decomposed hierarchically, and for each sub-task the agent retrieves execution-memory units whose embedding cosine similarity exceeds a predefined threshold $\\theta$. The agent then produces a solution in a fixed format that includes formulae, step-by-step reasoning, and executable Python code. An evaluate-and-refine module checks each sub-solution against knowledge memory, can restructure the remaining sub-task tree on failure, and successful sub-tasks are written back into execution and planning memory, so the library grows during inference.","core_discovery":"The paper's central claim is that decomposing chemistry problems into reusable atomic sub-tasks and storing those sub-tasks in a self-updating three-part library converts a static language model into an agent whose accuracy improves over iterations. The evidence is empirical: across the four datasets, the library-enhanced agent outperforms direct reasoning, code-based reasoning, and structured prompting, and the gap grows with the base model's strength. The authors also show that memory quality drives performance: memories produced by the stronger model beat memories produced by the weaker model, while mixing memory sources confuses the agent. In the self-evolution experiment, accuracy rises as correct solutions are added back into the library and converges above the static baseline, which the authors take as evidence that the library learns from practice.","pith_inferences":["The paper's retrieval failure suggests a condition-aware retriever as a direct next step: matching extracted conditions such as process type, units, and constants alongside embedding similarity would penalize adiabatic-versus-isothermal mismatches even when wording is close.","Because each sub-solution ends in executable Python, the library could double as a growing set of unit tests; re-running stored code on new numerical inputs would expose retrieved strategies that are internally inconsistent.","The self-evolution experiment only stores trials whose final answer is correct; testing whether storing refined-but-wrong attempts poisons the library would separate the benefit of accumulation from the benefit of filtering."],"forward_implications":["If ChemAgent is correct, an LLM can bootstrap its own domain expertise from a small solved development set and improve as it works, without retraining or hand-written demonstrations.","The reported gains are larger for stronger base models, so the same memory mechanism should become more valuable as the underlying models improve.","Memory quality is decisive: building the library with a weaker model lowers accuracy, and mixing memory from two different builders confuses the agent and hurts performance.","The evaluate-and-refine module contributes roughly five points on the strongest backbone but little on the weaker one, implying self-correction only pays off when the model is strong enough to act on feedback.","In the self-evolution run, accuracy converges above the no-memory baseline as correct solutions accumulate, implying the benefit is cumulative rather than a one-shot prompt effect."],"supporting_citations":[{"why":"Supplies the four college-chemistry datasets and the few-shot+Python baseline numbers used in all main experiments.","marker":"(Wang et al., 2024a)"},{"why":"Provides the structured-prompting baseline that ChemAgent is compared against and the source of the direct-reasoning baseline data.","marker":"(Ouyang et al., 2024)"},{"why":"The GPT-4 model used as the main reasoning backbone and as the generator of high-quality memory in the memory-quality experiments.","marker":"(OpenAI et al., 2024)"},{"why":"Provides the embeddings used for memory retrieval and the Llama3-7b and Llama3-70b backbones in the open-model experiments.","marker":"(Llama Team, 2024)"},{"why":"Motivates the least-to-most decomposition that the paper adapts into hierarchical sub-task splitting.","marker":"(Zhou et al., 2023)"},{"why":"Justifies ordering memory units from easy to hard during library construction.","marker":"(Bengio et al., 2009)"}],"fun_headline_variants":["ChemAgent's self-updating memory lifts LLM chemistry accuracy 46%","Reusable sub-tasks grant LLMs 46% better chemical reasoning","Self-evolving library boosts GPT-4 chemistry scores by 46%","LLM chemistry agent learns from memory, gains 46% accuracy","ChemAgent: Dynamic memory improves LLM chemistry reasoning by 46%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The retrieval step assumes that text similarity between a stored sub-problem and a new sub-problem is a reliable sign that the same chemical strategy will work; the paper's own error analysis shows that similar wording can hide a physically different process, and in that case the retrieved memory misleads instead of helping.","fun_headline_variants_meta":{"raw":{"variants":["ChemAgent's self-updating memory lifts LLM chemistry accuracy 46%","Reusable sub-tasks grant LLMs 46% better chemical reasoning","Self-evolving library boosts GPT-4 chemistry scores by 46%","LLM chemistry agent learns from memory, gains 46% accuracy","ChemAgent: Dynamic memory improves LLM chemistry reasoning by 46%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1196,"prompt_tokens":911,"completion_tokens":285,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":190}},"tokens_in":527,"tokens_out":285,"duration_ms":3311,"temperature":1.0,"reasoning_tokens":190,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:56:49.975567+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the agent two problems that read almost identically but differ in a condition that changes the governing formula, such as isothermal versus adiabatic compression; if the high-similarity memory is retrieved and its strategy is followed to the wrong answer even after the refinement step, the similarity proxy is the failure point. A systematic version would count how often accuracy drops when every test problem has a high-similarity distractor memory that shares vocabulary but requires a different formula.","supporting_citations":[],"review_version":1}