REVIEW 4 major objections 5 minor 6 cited by
ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoning
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read ChemAgent claims that a language model can solve multi-step chemistry problems more accurately when it builds and updates a structured library of previously solved sub-tasks.
desk verdict A useful agent framework for chemical reasoning, but the headline gains need re-measurement under a static library before they're trustworthy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a dynamic library with three memory types: planning memory stores high-level strategies; execution memory stores structured units of condition, sub-task, and sub-solution; and knowledge memory stores formulas and principles generated for the current problem. Sub-tasks are the atomic blocks: a new problem is decomposed hierarchically, and for each sub-task the agent retrieves execution-memory units whose embedding cosine similarity exceeds a predefined threshold $\theta$. The agent then produces a solution in a fixed format that includes formulae, step-by-step reasoning, and executable Python code. An evaluate-and-refine module checks each sub-solution against knowledge memory, can restructure the remaining sub-task tree on failure, and successful sub-tasks are written back into execution and planning memory, so the library grows during inference.
What would settle it
Give the agent two problems that read almost identically but differ in a condition that changes the governing formula, such as isothermal versus adiabatic compression; if the high-similarity memory is retrieved and its strategy is followed to the wrong answer even after the refinement step, the similarity proxy is the failure point. A systematic version would count how often accuracy drops when every test problem has a high-similarity distractor memory that shares vocabulary but requires a different formula.
Extended reading notes
Core claim
The paper's central claim is that decomposing chemistry problems into reusable atomic sub-tasks and storing those sub-tasks in a self-updating three-part library converts a static language model into an agent whose accuracy improves over iterations. The evidence is empirical: across the four datasets, the library-enhanced agent outperforms direct reasoning, code-based reasoning, and structured prompting, and the gap grows with the base model's strength. The authors also show that memory quality drives performance: memories produced by the stronger model beat memories produced by the weaker model, while mixing memory sources confuses the agent. In the self-evolution experiment, accuracy rises as correct solutions are added back into the library and converges above the static baseline, which the authors take as evidence that the library learns from practice.
Load-bearing premise
The retrieval step assumes that text similarity between a stored sub-problem and a new sub-problem is a reliable sign that the same chemical strategy will work; the paper's own error analysis shows that similar wording can hide a physically different process, and in that case the retrieved memory misleads instead of helping.
Editorial extensions
If this is right
- If ChemAgent is correct, an LLM can bootstrap its own domain expertise from a small solved development set and improve as it works, without retraining or hand-written demonstrations.
- The reported gains are larger for stronger base models, so the same memory mechanism should become more valuable as the underlying models improve.
- Memory quality is decisive: building the library with a weaker model lowers accuracy, and mixing memory from two different builders confuses the agent and hurts performance.
- The evaluate-and-refine module contributes roughly five points on the strongest backbone but little on the weaker one, implying self-correction only pays off when the model is strong enough to act on feedback.
- In the self-evolution run, accuracy converges above the no-memory baseline as correct solutions accumulate, implying the benefit is cumulative rather than a one-shot prompt effect.
Reading between the lines
- The paper's retrieval failure suggests a condition-aware retriever as a direct next step: matching extracted conditions such as process type, units, and constants alongside embedding similarity would penalize adiabatic-versus-isothermal mismatches even when wording is close.
- Because each sub-solution ends in executable Python, the library could double as a growing set of unit tests; re-running stored code on new numerical inputs would expose retrieved strategies that are internally inconsistent.
- The self-evolution experiment only stores trials whose final answer is correct; testing whether storing refined-but-wrong attempts poisons the library would separate the benefit of accumulation from the benefit of filtering.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. ChemAgent is an agentic framework for chemical reasoning that combines a structured library with three memory types (planning, execution, and knowledge) with LLM-based task decomposition, retrieval, and an evaluation-and-refinement module. The library is initially constructed from development-set problems by decomposing them into sub-tasks with conditions, questions, and solutions; during inference, new sub-tasks are matched to stored memory by Llama3-embedding cosine similarity, and the library is updated with newly solved sub-tasks and their solutions. On four SciBench chemistry datasets, the authors report that GPT-4-based ChemAgent reaches 57.16% average accuracy versus 47.66% for StructChem and 19.48% for direct reasoning, with a claimed gain of up to 46% on CHEMMC, and they report ablations of memory components, a memory-quality analysis, and a self-evolution experiment.
Significance. If the reported evaluation is valid, the contribution is significant: ChemAgent provides a concrete, fully prompted instantiation of memory-augmented decomposition for a domain where LLMs are known to make cascading errors, and the paper ships open-source code and detailed prompts in Appendix G. The ablations of memory components and memory quality, the cost analysis, and the self-evolution curve in Figure 5 are informative and go beyond a single accuracy number. However, the significance rests on the main comparison being apples-to-apples with static baselines, and that is not currently established because the runtime memory-update protocol exposes the test set to information that baselines never receive. The central claims of large gains over StructChem/direct reasoning and of self-evolution therefore need a clean static-versus-dynamic comparison before they can be taken at face value.
major comments (4)
- [§2.5, Fig. 2, Tables 1, 4, 5] The main comparison is confounded by transductive test-set exposure. Section 2.5 states that during library-enhanced reasoning "the memory is continuously updated with newly solved sub-tasks and their solutions during runtime," with Me = Me ∪ {(Cj,Tj,Oj)} and Mp updated similarly, and Figure 2 labels the pipeline as operating on the "test set" with an "(Updated) Library." This means that, under the protocol as written, an earlier test problem can seed memory used by later test problems. Section 2.9's exclusion rule only prevents a problem from being re-solved with memory derived from that same problem in a previous iteration; it does not prevent cross-problem information flow within the test set. With test sets of only 34-107 examples, a small number of cached solutions can move per-dataset accuracy by several percentage points. The baselines (Direct reasoning, Python, StructChem) never receive this exposure. Please report separately (i) a static-library run in which no test examples are written to memory and (ii) a self-evolving run with memory updates, and state explicitly which setting produced each row of Tables 1, 4, and 5.
- [§2.9, Fig. 5] The self-evolution experiment does not define how a solution is judged "correct" before being added to the library. Section 2.9 says that "all related responses and knowledge from that process are added to the library if the solution is correct," but no correctness-checking mechanism is described. If "correct" is determined by comparing against the test-set ground-truth answer, then the self-evolution curve in Figure 5 is fitted to the test set and its convergence to 44.89% is not evidence of generalizable self-improvement. Please specify the correctness criterion, and if it uses test labels, re-run with a label-free criterion or report both label-based and label-free versions.
- [Table 1, §2.7, §2.8] The main GPT-4 results lack error bars or any indication of run-to-run variance. In Table 1, differences such as ATKINS 61.18 vs. 59.81 (StructChem) and QUAN 44.12 vs. 41.18 (StructChem) are on the order of one or two test examples on datasets of 107 and 34 items, respectively. Appendix B states that open-source-model runs were repeated at least three times, but no variance is reported for GPT-4, and Table 1 gives single numbers. Please report repeated runs, seeds, and confidence intervals for the main comparisons, or at minimum report per-seed scores. Without this, the headline 9.5-point average gap cannot be separated from sampling noise.
- [§2.5, §2.11, Algorithm 1] The retrieval mechanism assumes that Llama3-embedding cosine similarity between sub-task texts tracks chemically meaningful problem structure, and the quality gate in Algorithm 1 assumes that LLM-assigned confidence correlates with factual correctness. Section 2.11 gives a concrete counterexample: a memory about entropy change during isothermal compression is retrieved for an adiabatic compression problem because the wording is similar, and the wrong strategy is used. The paper does not quantify how often such misleading retrievals occur or whether the confidence filter removes them. Because the library's benefit is the central claim, please report retrieval precision (for example, expert annotation of retrieved memories or a targeted error-case analysis) and the correlation between confidence scores and solution correctness.
minor comments (5)
- [Table 1 caption, §2.7] The caption says accuracy scores are computed using "the approximation detailed in Section 4.3," but Section 4.3 does not exist; the main text gives only the relative tolerance of 0.01. Please state the full metric definition in the main text and correct the reference.
- [Abstract, §2.8] The phrase "performance gains of up to 46%" is ambiguous: the underlying numbers are 28.21 vs. 74.36 on CHEMMC, which is an increase of 46 percentage points and a relative increase of roughly 163%. Please report the gain in consistent units.
- [§2.5, Algorithm 1, §2.7] Several hyperparameters are never given numerical values: the similarity threshold θ, the LLM-confidence threshold for discarding memory units, the maximum number of refinement operations, the accuracy tolerance, and the top-k for imagined problems. If these are set in the released code, please state their values or point to the exact configuration file in the main text.
- [Appendix B, Table 5] The Table 5 title reads "Llama 3.1-70b, Llama 3.1-70b and Qwen 2.5-72b," which appears to duplicate the Llama 3.1-70b entry; the intended list is likely Llama 3.1-7b, Llama 3.1-70b, and Qwen 2.5-72b.
- [§2.8] The sentence "improves by 9.50% (47.66 vs. 57.16) over StructChem, which is a 2.93 times increase and by 37% (19.48 vs. 57.16) over direct reasoning" mixes percentage-point gains with relative increases. Please rephrase to distinguish absolute percentage-point changes from multiplicative improvements.
Circularity Check
Test-set memory updates make the headline gains partly transductive; the self-evolution experiment explicitly populates the library from the test set, so its 'improvement' is forced by the protocol rather than demonstrated.
-
fitted input called prediction
[Section 2.5, Library-Enhanced Reasoning (runtime update equations) with Table 1]
"Moreover, the memory is continuously updated with newly solved sub-tasks and their solutions during runtime. For each newly solved sub-task Tj, sub-answers from previous sub-problems are incorporated into the conditions as Cj, and the execution memory is updated based on solution Oj: Me = Me ∪ {(Cj, Tj, Oj)}."
During test-set inference, the library is populated from the test problems themselves, and then Table 1 reports ChemAgent's accuracy on that same test set. The static baselines (Direct reasoning, Python, StructChem) receive no such test-set exposure, so the reported gains (e.g., CHEMMC 74.36 vs 28.21, average 57.16 vs 47.66) can reflect transductive memorization of earlier test problems rather than a generalizable library effect. The paper does not provide a static-library-only run for Tables 1, 4, or 5, so the headline 'prediction' is not cleanly separated from its test-set input.
-
fitted input called prediction
[Section 2.9, Self-Evolution During Runtime, and Figure 5]
"We allow ChemAgent to dynamically update and enrich its library during the test stage to analyze this self-evolution process. ... When solving a new problem P, all related responses and knowledge from that process are added to the library if the solution is correct. ... To ensure accuracy and prevent target leakage, the memory derived from problem Pi in iteration Ij is excluded when solving Pi again in Ik for any j < k."
The self-evolution improvement in Figure 5 is measured on the test set while the memory is built from that same test set, filtered by the ground-truth correctness of the test solutions. Excluding only the same problem's memory still permits all other test problems' memories to be reused, so the library is effectively fit to the evaluation distribution. The observed convergence above the static baseline is therefore a transduction artifact of the protocol, not evidence of generalizable self-evolution on unseen problems.
full rationale
The core accuracy metric is external (SciBench test answers), so the derivation chain is not circular in the sense that a formula is assumed in its own output. Likewise, the self-citations (notably StructChem, by co-author Ouyang et al.) are used as a comparison baseline and as a statement that SciBench is representative; they are not load-bearing for the framework's internal mechanism. However, the method as described updates its memory from the test set during inference (Section 2.5: Me = Me ∪ {(Cj, Tj, Oj)}), and the paper does not report a static-library-only configuration for the main tables. The self-evolution section is more explicit: memory is added only 'if the solution is correct,' i.e., using test-set labels to select the library, while only the same problem's memory is excluded across iterations. This leaves cross-problem test-set leakage intact. Consequently, the reported performance gains and the convergence claim are partially forced by the evaluation protocol rather than by independent generalization. The LLM-generated quality gates (Algorithm 1's confidence filtering and the Evaluation & Refinement module's self-scoring) are self-referential, but because final accuracy is judged against external ground truth, they are not themselves circular; they are a correctness-risk issue. On balance, this is partial circularity of the evaluation claim: score 6.
Assumptions & free parameters
free parameters (6)
- similarity threshold theta =
not reported; 0.805 used in analysis
- retrieved memory counts (Mp, Me) =
2-shot, 4-shot
- memory confidence threshold =
not specified
- max refinement operations =
4
- accuracy tolerance =
0.01 relative
- top-k imagination problems =
not specified
assumptions (4)
- domain assumption Cosine similarity between Llama3 embeddings of sub-task texts is a reliable proxy for chemical problem-structure similarity.
- domain assumption The development set and its reference solutions are representative of test-set sub-tasks.
- ad hoc to paper LLM-generated memory units are chemically reliable after LLM-based confidence filtering.
- domain assumption Sub-task decompositions align one-to-one with sub-solutions and preserve solution correctness.
invented entities (1)
-
Synthetic execution memory
Cite this review
Pith. "Pith review of ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoning." pith.science (2026). https://pith.science/paper/ZKO5XE7Q
@misc{pith2026250106590,
author = {Pith},
title = {Pith review of: ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoning},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZKO5XE7Q}},
note = {Machine review of arXiv:2501.06590}
}
read the original abstract
Chemical reasoning usually involves complex, multi-step processes that demand precise calculations, where even minor errors can lead to cascading failures. Furthermore, large language models (LLMs) encounter difficulties handling domain-specific formulas, executing reasoning steps accurately, and integrating code effectively when tackling chemical reasoning tasks. To address these challenges, we present ChemAgent, a novel framework designed to improve the performance of LLMs through a dynamic, self-updating library. This library is developed by decomposing chemical tasks into sub-tasks and compiling these sub-tasks into a structured collection that can be referenced for future queries. Then, when presented with a new problem, ChemAgent retrieves and refines pertinent information from the library, which we call memory, facilitating effective task decomposition and the generation of solutions. Our method designs three types of memory and a library-enhanced reasoning component, enabling LLMs to improve over time through experience. Experimental results on four chemical reasoning datasets from SciBench demonstrate that ChemAgent achieves performance gains of up to 46% (GPT-4), significantly outperforming existing methods. Our findings suggest substantial potential for future applications, including tasks such as drug discovery and materials science. Our code can be found at https://github.com/gersteinlab/chemagent
Figures
Figures from the paper (16 more)
Forward citations
Cited by 6 Pith papers
-
LatentChem: From Textual CoT to Latent Thinking in Chemical Reasoning
LatentChem reasons in continuous latent space for chemistry, achieving a 59.88% non-tie win rate over explicit CoT on ChemCoTBench with a 10.84x average reduction in reasoning overhead.
-
Experience-Evolving Multi-Turn Tool-Use Agent with Hybrid Episodic-Procedural Memory
An LLM agent that builds a tool-transition graph with state summaries from past experience improves tool selection and RL exploration by large margins on multi-turn benchmarks.
-
G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems
G-Memory stores past multi-agent teamwork in a three-tier graph and retrieves it to boost performance on five benchmarks.
-
AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes
AgentDistill distills agent capabilities without any training by having a teacher generate reusable MCP tool boxes that small-model students invoke at inference time.
-
AutoMat: Enabling Automated Crystal Structure Reconstruction from Microscopy via Agentic Tool Use
AutoMat reconstructs crystal structures from simulated STEM images by orchestrating denoising, template retrieval, symmetry-constrained atom fitting, and machine-learned-potential relaxation, and it outperforms existi...
-
El Agente: An Autonomous Agent for Quantum Chemistry
A hierarchical multi-agent LLM system autonomously plans, executes, and debugs quantum chemistry calculations, achieving over 87% success on six benchmark exercise types and two case studies.
Reference graph
Works this paper leans on
-
[2]
provides structured guidance by decomposing chemical reasoning into phases such as formula generation, detailed step-by-step reasoning, and confidence-based review. While showing improve- ments, it still faces limitations with highly complex problems. Other researchers have explored enhancing LLM performance through various prompting strategies (Yang et a...
work page 2023
-
[3]
Handle subtasks recursively until all subtasks are solved. Do not make the task queue too complex, make it efficiently solve the original task. RESOURCES: A task-handling agent can write and execute Python code. — Task Description — Generate the plan for query with operation SUBTASKSPLIT, and make sure all must-reach goals are included in the plan. — Impo...
-
[5]
Convert volume to liters (since 1 m3 = 1000 L): V = 8.6801 × 10−6 m3 × 1000 L/m3 = 0.0086801 L Execution of Task 2 - Convert the V olume to cm3 Formula: - Conversion from liters to cubic centimeters: 1 L = 1000 cm3 Calculation:
-
[6]
Explicitly include all the formulas used during performing the subtasks in this summary
Summary: Summarize the ideas of all subtasks to the parent task, and summarize a total process to the parent task according to the action process of each subtask. Explicitly include all the formulas used during performing the subtasks in this summary. Specific numbers and numerical results are NOT needed in this part
-
[7]
Reflection of knowledge: After performing this task, you get some knowledge of generating plans for similar tasks. Only conclude the used knowledge and formulas used in this whole task, do NOT contain numerical calculation process and results
-
[8]
Final Answer: Give the final answer to the task according to the course of the task, and ask the answer to be very short, without explaining the reason and adding unnecessary punctuation. If it’s a math problem, only the last value is given. Figure 17: A prompt for summarizing tasks. Summarizing tasks is crucial for the self-evolution of ChemAgent. Summar...
-
[9]
it will handle subtask 1 first
-
[10]
If failed, split subtask 1 as subtask 1.1 1.2 1.3
if solved, handle subtask 2. If failed, split subtask 1 as subtask 1.1 1.2 1.3... Then handle subtask 1.1 1.2 1.3
Show all 26 references
-
[11]
Do not make the task queue too complex, make it efficiently solve the original task
Handle subtasks recursively until all subtasks are solved. Do not make the task queue too complex, make it efficiently solve the original task. RESOURCES: A task-handling agent can write and execute Python code. — Task Description — Your task is iteratively to rectify a given ...
-
[12]
target subtask id
split: Split an already handled but failed subtask into subtasks because it is still so hard. The “target subtask id” for this operation must be a leaf task node that has no children subtasks, and should provide new split “subtasks” of length 2-4. You must ensure the “target s...
-
[13]
This operation will expand the width of the plan tree
add: Add new subtasks as brother nodes of the ‘target subtask id‘. This operation will expand the width of the plan tree. The ‘target subtask id‘ should point to a now-handling subtask or future subtask. - add 1.1 with two subtasks will result in creating new 1.2, 1.3 subtasks...
-
[14]
find A,B,C
delete: Delete a subtask. The ‘target subtask id‘ should point to a future/TODO subtask. Don’t delete the now handling or done subtask. - delete 1.2.1 will result in delete 1.2.1 subtask. — Note — The user is busy, so make efficient plans that can lead to successful task-solvi...
-
[15]
Figure 18: A prompt for refining tasks
Please use a function call to respond to me (remember this!!!). Figure 18: A prompt for refining tasks. Incorrect answers may result from an error in a sub-question, leading to subsequent inaccuracies, or from a misjudgment in the decomposition of the question. In the former c...
2024
-
[16]
Convert pressure from atm to Pa: P = 20 atm × 101325 Pa/atm = 2, 026, 500 Pa
-
[17]
Calculate the volume in cubic meters: V = 0.86 × 8.2 × 10−3 mol × 8.3145 J K−1 mol−1 × 300 K 2, 026, 500 Pa
Rearrange the compression factor formula to solve for V : Z = P V nRT =⇒ V = ZnRT P3. Calculate the volume in cubic meters: V = 0.86 × 8.2 × 10−3 mol × 8.3145 J K−1 mol−1 × 300 K 2, 026, 500 Pa
-
[18]
Compute V : V = 0.86 × 8.2 × 10−3 × 8.3145 × 300 2, 026, 500 V ≈ 8.6801 × 10−6 m3
-
[20]
We also provide an execution trajectory of a successfully answered question in another benchmark, ChemBench (Mirza et al., 2024) which direct-reasoning method fails to solve
Convert volume from liters to cubic centimeters: V = 0.0086801 L × 1000 cm3/L = 8.6801 cm3 Summary Results The volume occupied by 8.2 mmol of the gas under the given conditions is approximately 8.6801 cm3. We also provide an execution trajectory of a successfully answered ques...
2024
-
[21]
Ppartial = ngasRT V
The partial pressure of a gas in a container is given by the ideal gas law, which states that the partial pressure is directly proportional to the number of moles of the gas, the temperature, and inversely proportional to the volume. Ppartial = ngasRT V
-
[22]
Therefore, the partial pressure of gas (X) remains the same because ngas, R, T , and V are constant
When a non-reacting gas is added to the container at constant volume and constant temperature, the number of moles of gas (X) does not change. Therefore, the partial pressure of gas (X) remains the same because ngas, R, T , and V are constant. The initial and final partial pre...
-
[23]
The initial total pressure is the partial pressure of gas (X), PX
Consider a container with gas (X) at constant volume and temperature. The initial total pressure is the partial pressure of gas (X), PX
-
[24]
When a non-reacting gas (Y) is added to the container at constant volume and temperature, the partial pressure of gas (X) remains the same, PX
-
[25]
Execution of Task 3 - Compare Statements and Choose the Correct Option Formula: No specific formula is needed for this subtask
The total pressure of the container after adding gas (Y) is the sum of the partial pressures of gas (X) and gas (Y), Ptotal = PX + PY . Execution of Task 3 - Compare Statements and Choose the Correct Option Formula: No specific formula is needed for this subtask. 34 Reasoning:
-
[26]
- From Task 1.2: The total pressure increases after adding a non-reacting gas at constant volume and temperature, but the partial pressure of gas (X) remains unchanged
Analyze the results from the previous subtasks: - From Task 1.1: The partial pressure of gas (X) remains unchanged after adding a non-reacting gas at constant volume and temperature. - From Task 1.2: The total pressure increases after adding a non-reacting gas at constant volu...
-
[27]
Compare the findings from the analysis of Statements 1 and 2
-
[28]
Summary Results D
Determine the most correct option based on the analysis: - Both statements are true, and Statement 2 correctly explains Statement 1. Summary Results D. Statements 1 and 2 are True. Statement 2 is the correct explanation for Statement 1. An example result of the decomposition s...
-
[2024]
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, et al
URL https://arxiv.org/abs/2404.01475. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, et al. GPT-4 technical report, 2024. Magda Osman. An evaluation of dual-process theories of reasoning.Psychonomic bulletin & review, 11(6):988–1010, 2004. Siru Ouyang, Zhuosh...
2024 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.