Artificial Intelligence · arXiv:2610.34203 v2 · submitted 25 Oct 2026
MatOS: An Agent Operating System for Materials Research
The paper says
“Materials research relies on scattered and growing data, knowledge, and simulation tools, and a single study often spans multi-day workflows across literature analysis, computation, and experiment.”
more of the abstract
Large language model agents can now retrieve materials data, operate simulation codes, and close design-synthesis-characterization loops with autonomous laboratories. However, extending these agents with new capabilities, keeping long-running tasks on track, and letting them learn from past tasks and user feedback remain difficult. Here we present MatOS, an agent operating system for materials research with three core components: (1) a layered scientific asset graph that organizes data, knowledge, tools, skills, roles, and workflows as reusable assets; (2) a stateful adaptive workflow that runs tasks as explicit execution graphs with scientific checks, so that they can be resumed and traced; and (3) evidence-driven asset evolution, which revises these assets from execution records and user feedback. On ResearchClawBench, MatOS improves on the previously best-performing agent by 11.15% with the same base model and extends the cost-score Pareto frontier. With a common base model, it outperforms Codex CLI and Claude Code, exceeding Codex CLI by 2.13% on MatTools and 4.17% on SciAgentGym. In eight research cases, MatOS narrowed more than 30 million candidate structures to 97 solid-electrolyte candidates, identified 89 halide variants with higher computed conductivity than their parent structures, linked the force errors of 11 pretrained interatomic potentials to local atomic environments, and supported two new methods that reduce errors by 3.1% and 13.6% relative to their baselines. These results show that organizing research capabilities as reusable, evolving assets enables agents to carry out long, multistage materials studies. Code is available at https://github.com/DingyangLyu/MatOS.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 27 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
The passes disagreed. Reader: holds up. Skeptic: holds up only with conditions. Referee: major revision.
A solidly engineered system with good benchmark results, but the research cases are more human-led than agent-autonomous, undercutting the core claim. This line is wrong
3 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Section 4 and Contributors
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The Contributors section states that Hongwei Du 'carried out six of the eight research cases' (Sections 4.1–4.3 and 4.4.2). The paper presents these cases as demonstrations of MatOS capabilities (b9: 'enables agents to carry out long, multistage materials studies reliably'), but it is not clear what 'carried out' means.
more of objection 1
If the human researcher defined the research direction, selected methods, and interpreted intermediate results, while the agent merely executed predefined workflows, then the cases do not demonstrate agent autonomy. The authors must specify for each research case which steps were autonomously executed by the MatOS agent and which required human intervention. Provide a detailed breakdown (e.g., decision nodes, method selection, result interpretation) to distinguish agent actions from human actions. Without this, the central claim that MatOS itself enables reliable long-horizon studies is unsupported by the case-study evidence.
The paper says, at Section 4.1.2 (Training-Data Selection)
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.61):
“The evaluation sets were also used during selector development.”
Pith says: Objection 2 of 3 · major
The paper acknowledges (b267) that 'The evaluation sets were also used during selector development.' This raises a concern of data leakage or overfitting: the training-data selection method RarityGuard-MP was developed using the same evaluation sets on which it is tested.
more of objection 2
The reported 3.1% aggregate improvement over DIRECT may therefore be optimistic. The authors should clarify exactly which evaluation sets were used during development (e.g., for hyperparameter tuning or method decisions) and, if any overlap exists, provide additional validation on a held-out set or discuss the potential bias. This is a standard methodological requirement for any comparative evaluation.
The paper says, at Section 2.3 and 4
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The evidence-driven asset evolution component is a key architectural contribution, but its role in the research cases is unclear. The paper states that the asset library was frozen in all benchmark evaluations (Section 3.2). The research cases are the only setting where evolution could occur, but the Contributor section implies heavy human involvement.
more of objection 3
The authors should clarify whether the asset evolution mechanism was activated during the research cases, and if so, provide concrete examples of asset revisions (e.g., a skill change, a tool update) that were proposed, checked, and adopted autonomously by the system. Without such examples, the claim that 'experience gained in one task can be reused in later ones' remains unvalidated.
6 minor objections
The paper says, at Abstract
Pith says: minor objection 1 of 6
The abstract states 'MatOS improves on the previously best-performing agent by 11.15%' without specifying the benchmark. Add 'on ResearchClawBench' for clarity.
The paper says, at Section 3.3, Figure 6
Pith says: minor objection 2 of 6
The cost axis is logarithmic, but the non-dominated front (teal dashed line) appears to be drawn linearly. Clarify in the caption whether the front is linear in log-space or actual cost. Also check the interpolation between points.
show 4 more
The paper says, at Section 2.3, first sentence of 'Evidence critique'
Pith says: minor objection 3 of 6
The sentence 'User corrections, failed tool calls, repeated operations, task summaries, and workflow results provide candidate learning signals.' could benefit from a brief example to illustrate what constitutes a 'candidate learning signal' in a materials context.
The paper says, at Table 5
Pith says: minor objection 4 of 6
MatOS is marked with checkmarks for 'Experience extraction' and 'Update acceptance'. However, in the experimental evaluation, self-evolution was disabled. Add a footnote to the table noting that these capabilities were not evaluated in the benchmarks and were only illustrated in the research cases.
The paper says, at Appendix C
Pith says: minor objection 5 of 6
The relative L2 differences between the OpenFOAM solution and the stored CFDBench fields (8.9%, 8.1%, 7.6%) are relatively large. The paper attributes this to interpolation and discretization error of the stored fields, but a brief quantification of the expected interpolation error would strengthen the discussion.
The paper says, at References
Pith says: minor objection 6 of 6
Some references list all authors (e.g., [38]), while others use et al. after six or more authors (e.g., [6]). Standardize the style to one convention.
What would settle it
Re-run one of the six human-led research cases (e.g., Section 4.2.1 solid-electrolyte screening) in a fully autonomous mode: provide only the initial task description and the frozen asset library (as in benchmarks). Log every agent decision (tool selection, parameter choice, filter criterion) and every human intervention. Report the fraction of steps executed without human input and whether the final result (97 candidates) is reproduced. If the agent cannot independently complete the pipeline or requires frequent intervention, the central claim is weakened accordingly.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 27 Oct 2026
- Reader holds up
- Skeptic partly agrees: says it holds up only with conditions Research cases are human-led, not agent-autonomous, undermining the claim that MatOS enables agents to carry out long studies independently.
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
You should know two things about this paper. First, the architecture is genuinely well thought out: the layered asset graph, the check-compiled workflows, and the evidence-driven revision cycle are a clean integration of ideas from scientific workflows, agent harnesses, and self-improvement. The benchmark numbers are consistent and non-trivial—11% on ResearchClawBench with the same base model, and small but clean gains on MatTools and SciAgentGym. The system exists, the code is out, and the evaluation is unusually thorough.
Second, the stress-test hits a real soft spot. The paper's headline claim—that MatOS "enables agents to carry out long, multistage materials studies reliably"—rests heavily on the eight research cases. Those cases are presented as demonstrations of agent capability, but the Contributor section tells a different story: Hongwei Du personally carried out six of the eight. The reader cannot tell which steps were agent-autonomous and which required human judgment. When you read the case write-ups, the language is passive ("MatOS screens…", "MatOS compares…"), but the actual execution clearly involved substantive human direction. That is a mismatch between evidence and conclusion. The benchmark evaluations do test autonomous performance, but they are limited to predefined tasks with no asset evolution, so they don't support the long-horizon autonomy claim either.
That said, the paper acknowledges that reliable agent behavior remains an open problem, and the system architecture itself does not depend on perfect autonomy—it is designed to incorporate human checkpoints. The fault is in the framing, not in the engineering. If the authors revised the conclusion to say "MatOS enables researchers to carry out long, multistage materials studies by organizing capabilities as reusable assets," the evidence would support it cleanly.
For a reading group focused on AI for science, this is worth a session—it is a concrete system with real data, and the autonomy question is exactly the kind of thing worth debating. I would not cite it for the claimed autonomous studies, but I might cite the architecture or the benchmark results. I would send it to review, with a note to the authors that they need to clarify human involvement in the cases and adjust the claims accordingly.
The strongest objection
The central claim (b9) states that organizing capabilities as reusable assets "enables agents to carry out long, multistage materials studies reliably." The strongest real-world evidence is the eight research cases (Section 4). However, the Contributor section explicitly states that Hongwei Du "carried out six of the eight research cases" (Sections 4.1–4.3 and 4.4.2). This means the scientific direction, method selection, and interpretation were primarily human-driven, not autonomously executed by a MatOS agent. The paper presents these cases as demonstrations of MatOS capabilities, but the reader cannot distinguish which steps were performed by the agent versus by the human author. For example, the solid-electrolyte screening (Section 4.2) and the alloy composition search (Section 4.4.1) involve complex multi-stage pipelines; it is unclear whether the MatOS agent autonomously composed the tools and skills or whether the human researcher manually orchestrated the workflow. If the cases required substantial human intervention (e.g., defining filters, selecting models, interpreting intermediate results), then they do not support the claim that the agent system itself can carry out such studies reliably. The benchmark evaluations (ResearchClawBench, MatTools, SciAgentGym) do test autonomous agent performance, but these are limited to predefined tasks with a frozen asset library and no asset evolution. The asset evolution and long-horizon autonomy are only showcased in the cases, which are compromised by human leadership. The paper does not report any measure of autonomy (e.g., fraction of decisions made by the agent vs. human) for the cases, leaving a critical gap between evidence and conclusion.
The reader's weakest_assumption identified that LLM agents may not effectively select and compose assets autonomously. My concern directly operationalizes that assumption: the primary evidence of long-horizon capability (eight research cases) is tainted by human leadership, so the assumption remains untested for the most demanding scenarios. The paper acknowledges that 'reliable agent behavior remains an open challenge' (b23, b25), but then offers case studies that implicitly claim reliability without documenting autonomy. This is not a fatal flaw; the benchmarks do demonstrate benefit on shorter tasks. However, to accept the paper's central conclusion about long, multistage studies, the community needs to know whether the cases were agent-run or human-run. A condition requiring explicit reporting of human vs. agent contributions would resolve the ambiguity and strengthen the paper's contribution.
The referee's report
If the central claim holds, MatOS represents a significant architectural contribution to AI-driven materials research. The layered scientific asset graph, stateful adaptive workflow, and evidence-driven asset evolution collectively address three persistent challenges: extensibility, long-running execution reliability, and experience accumulation. The benchmark results are convincing, with improvements on three independent benchmarks, including a double-digit gain on ResearchClawBench. The paper also provides extensive supplementary material (asset library composition, workflow examples, an appendix with a fluid-dynamics reproduction) that supports reproducibility. The approach is general and could influence agent system design in other scientific domains.
Pith's reading of the paper (a machine's summary, not the authors')
MatOS is an agent operating system designed to solve three persistent challenges in materials research automation: adding new capabilities without rebuilding the agent, keeping long-running multi-day tasks on track, and learning from past experience. It treats data, knowledge, tools, skills, roles, and workflows as a layered graph of reusable scientific assets, runs each task as an explicit execution graph with scientific checks and persistent state, and revises assets from execution traces and user feedback after each task. On ResearchClawBench, with the same base model, MatOS improves over the previously best-performing agent by 11.15% and extends the cost-score Pareto frontier. It also outperforms Codex CLI and Claude Code on materials tool use and scientific problem-solving. Beyond benchmarks, it screened over 30 million candidate structures to find 97 solid-electrolyte candidates, identified 89 halide variants with higher conductivity than their parents, and helped develop two new methods that reduce errors relative to their baselines.
What carries the argument. The key machinery is the layered scientific asset graph, which defines six asset types (data, knowledge, tools, skills, roles, workflows) and six directed relation types (dependency, mount, call, containment, evidence, verification). This graph forms the substrate that the stateful adaptive workflow runtime and evidence-driven asset evolution operate on: the workflow composes assets for each task and records execution state, while the evolution mechanism locates and revises the assets responsible for observed problems.
Core claim. The core discovery is that organizing research capabilities as reusable scientific assets—each with a unique identifier, declared dependencies, and provenance—enables agents to compose and extend capabilities for new tasks without retraining or rebuilding. The layered asset graph with six types (data, knowledge, tools, skills, roles, workflows) and six typed relations (dependency, mount, call, containment, evidence, verification) captures how research methods depend on each other and on evidence. The stateful adaptive workflow executes research tasks as explicit graphs with checked plan updates and persistent state, allowing long-running studies to be paused, resumed, and traced. Evidence-driven asset evolution critiques execution traces and user feedback, then revises assets through structural checks and semantic review. Together, these components yield measurable improvements in end-to-end research quality, tool use, and scientific problem-solving compared to previous agent systems.
The premise it rests on. The system assumes that large language model agents can reliably select and combine the appropriate scientific assets from the layered graph for any given materials research task, and that the six asset types and six relation types are sufficient to represent all necessary research capabilities.
If the paper is right (Pith's inferences, not the authors')
New materials databases, simulation codes, and machine-learning potentials can be added by registering assets in the graph rather than writing new tool functions or modifying pipelines.
Experience gained in one research task—including failure diagnoses, validation criteria, and method refinements—can be checked and reused in later tasks through asset evolution, without retraining the underlying language model.
Long-running, multi-day studies become more reliable because the execution graph persists through interruptions and plan changes, and only proposals that pass structural, policy, and compilation checks are admitted.
The architecture extends directly to experimental instruments and autonomous laboratories, allowing a single study to link computational and experimental evidence within the same asset framework.
Other scientific fields with similarly heterogeneous and expanding tool ecosystems (biology, chemistry, fluid dynamics) could adopt the same asset-based organization and workflow runtime.
By turning capability extension into a curation problem (registering and checking assets rather than writing code), MatOS lowers the software-engineering barrier for domain scientists to contribute new methods and data sources to the agent system.
The dual-acceptance process for asset revisions (structural checks plus semantic review that verifies no new errors) sets a higher bar for autonomous improvement than most existing self-evolution frameworks, which typically rely solely on task-success statistics.
The explicit representation of checking and validation procedures as assets (guard nodes, verification relations) addresses a known gap in AI reliability: that repeated runs of the same agent can produce different outcomes, and that the system must determine which earlier results remain valid after a plan change.
The case study of the CFDBench geometry-subset metadata inconsistency (a lid-speed error in the benchmark) illustrates that agent systems can also serve as auditing tools for scientific benchmarks themselves.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Materials research relies on scattered and growing data, knowledge, and simulation tools, and a single study often spans multi-day workflows across literature analysis, computation, and experiment. Large language model agents can now retrieve materials data, operate simulation codes, and close design-synthesis-characterization loops with autonomous laboratories. However, extending these agents with new capabilities, keeping long-running tasks on track, and letting them learn from past tasks and user feedback remain difficult. Here we present MatOS, an agent operating system for materials research with three core components: (1) a layered scientific asset graph that organizes data, knowledge, tools, skills, roles, and workflows as reusable assets; (2) a stateful adaptive workflow that runs tasks as explicit execution graphs with scientific checks, so that they can be resumed and traced; and (3) evidence-driven asset evolution, which revises these assets from execution records and user feedback. On ResearchClawBench, MatOS improves on the previously best-performing agent by 11.15% with the same base model and extends the cost-score Pareto frontier. With a common base model, it outperforms Codex CLI and Claude Code, exceeding Codex CLI by 2.13% on MatTools and 4.17% on SciAgentGym. In eight research cases, MatOS narrowed more than 30 million candidate structures to 97 solid-electrolyte candidates, identified 89 halide variants with higher computed conductivity than their parent structures, linked the force errors of 11 pretrained interatomic potentials to local atomic environments, and supported two new methods that reduce errors by 3.1% and 13.6% relative to their baselines. These results show that organizing research capabilities as reusable, evolving assets enables agents to carry out long, multistage materials studies. Code is available at https://github.com/DingyangLyu/MatOS.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-27
The authors' main claim, as the classifier picked it out 1.00
“These results show that organizing research capabilities as reusable, evolving assets enables agents to carry out long, multistage materials studies.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.19 · human participants 0.10 · controlled comparison 0.87 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.98
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Materials research relies on scattered and growing data, knowledge, and simulation tools, and a single study often spans multi-day workflows across literature analysis, computation, and experiment. | background 1.00 | 0.05 | 0.02 | asserted 1.00 |
| Large language model agents can now retrieve materials data, operate simulation codes, and close design-synthesis-characterization loops with autonomous laboratories. | background 1.00 | 0.06 | 0.03 | asserted 0.82 |
| However, extending these agents with new capabilities, keeping long-running tasks on track, and letting them learn from past tasks and user feedback remain difficult. | gap 0.98 | 0.06 | 0.02 | asserted 1.00 |
| Here we present MatOS, an agent operating system for materials research with three core components: (1) a layered scientific asset graph that organizes data, knowledge, tools, skills, roles, and workflows as reusable assets; (2) a stateful adaptive workflow that runs tasks as explicit execution graphs with scientific checks, so that they can be resumed and traced; and (3) evidence-driven asset evolution, which revises these assets from execution records and user feedback. | contribution 0.91 | 0.85 | 0.10 | asserted 1.00 |
| On ResearchClawBench, MatOS improves on the previously best-performing agent by 11.15% with the same base model and extends the cost-score Pareto frontier. | result 1.00 | 0.95 | 0.08 | asserted 1.00 |
| With a common base model, it outperforms Codex CLI and Claude Code, exceeding Codex CLI by 2.13% on MatTools and 4.17% on SciAgentGym. | result 1.00 | 0.93 | 0.04 | asserted 1.00 |
| In eight research cases, MatOS narrowed more than 30 million candidate structures to 97 solid-electrolyte candidates, identified 89 halide variants with higher computed conductivity than their parent structures, linked the force errors of 11 pretrained interatomic potentials to local atomic environments, and supported two new methods that reduce errors by 3.1% and 13.6% relative to their baselines. | result 1.00 | 0.93 | 0.08 | asserted 1.00 |
| These results show that organizing research capabilities as reusable, evolving assets enables agents to carry out long, multistage materials studies. | interpretation 1.00 | 0.66 | 0.91 | asserted 1.00 |
| Code is available at https://github.com/DingyangLyu/MatOS. | contribution 0.59 | 0.42 | 0.02 | asserted 0.74 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from simulation or computation 0.90
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-27: 1 request, 7,186 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (21)
Figures are the authors', reproduced from the paper.
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment