{"id":"18b3a71b-408e-474b-ad59-b2467d060c5e","arxiv_id":"2507.08958","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A multi-agent large language model system extracts cosmological simulation parameters from published papers and generates valid MP-GADGET configuration files, achieving 98.7% micro-F1 on a new 41-simulation benchmark.","lead":"SimAgents uses two specialized AI agents to read cosmology papers and simulation software manuals, turning published simulation setups into ready-to-run configuration files in about two minutes. If the reported accuracy holds up, it could cut a common, error-prone workflow for cosmologists from hours to minutes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline error-reduction claim (80% vs CoT, 70% vs EoT) is not derivable from the paper's own Table 2; per-simulation totals give roughly 54% and 56% reductions when hallucination errors are included.","rationale":"The reader's single-annotation concern is real and is acknowledged in the paper's limitations, but a more immediate, internally checkable inconsistency appears in the headline quantitative claim. The 98.67 micro-F1 is supported by both the human evaluation (Table 1) and the automated evaluation against the annotated dataset (Appendix C, 98.13), so the F1 component is less fragile. The 80%/70% error-reduction figure, by contrast, is not derivable from Table 2 under any natural reading of the reported per-simulation error averages. This is a correctness risk in a central result of the paper. The fix is a definition and recomputation, so the conditional verdict remains appropriate; the released code and dataset are positive factors that make verification feasible. The concern is about the reported arithmetic and metric definition, not about the authors' intent or conduct.","tokens_in":10255,"tokens_out":7547,"duration_ms":84196,"concrete_test":"Recompute the error-reduction percentages from Table 2 and Table 4 under each plausible definition of \"overall error rate\" (all error types, value+type only, value only, per simulation vs per parameter). Specifically, sum Table 2 rows and calculate (baseline_total - ours_total) / baseline_total. If no definition yields 80% for CoT and 70% for EoT simultaneously, the claim must be corrected or the tables reconciled; if a definition is intended, it must be stated in the text and the applicable table cited.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states that SimAgents \"reduces the overall error rate by 80% compared to CoT and 70% compared to EoT.\" Table 2 reports average errors per simulation by type. Summing all three error types: CoT = 0.97 + 0.51 + 0.21 = 1.69; EoT = 1.21 + 0.21 + 0.34 = 1.76; SimAgents = 0.46 + 0.02 + 0.30 = 0.78. The reductions are (1.69 - 0.78)/1.69 = 53.8% and (1.76 - 0.78)/1.76 = 55.7%, not 80% and 70%. Excluding hallucination errors gives 67.6% and 66.2%. The automated Appendix C table, which has no hallucination column, gives roughly 84% for both baselines. No consistent definition of \"overall error rate\" in the paper reproduces the stated 80%/70% pair. Since this quantitative claim is part of the paper's strongest result, it is load-bearing: the claimed improvement is not reproducible from the paper's own tables. The authors should define the metric and correct either the percentages or the tables. This does not necessarily invalidate the system, but the central claim as written is unsupported.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SimAgents, a multi-agent large-language-model system that automates the extraction of cosmological simulation parameters from published papers and generates executable MP-GADGET configuration files, together with preliminary post-processing analysis code. The system uses a Physics Agent and a Software Agent in iterative discussion, where the Physics Agent reads the paper and the Software Agent enforces the user-manual constraints. The authors construct a benchmark dataset of 41 simulations from published cosmology papers, manually annotated with MP-GADGET parameter pairs, and evaluate SimAgents against chain-of-thought (CoT) and exchange-of-thought (EoT) baselines. The main reported results are a Micro-F1 of 98.67%, precision 97.80%, and recall 99.55% for SimAgents (Table 1), and a claim that SimAgents reduces the overall error rate by 80% relative to CoT and 70% relative to EoT (Section 4.1). The paper also includes ablation studies on the number of discussion rounds and on a smaller backbone model Qwen3-4B, plus a short demonstration of post-simulation analysis code generation.","tokens_in":10544,"tokens_out":4306,"duration_ms":48862,"significance":"If the reported results hold, SimAgents would be a practically useful tool for cosmologists, converting dense literature descriptions into simulation-ready configuration files in minutes rather than hours, and the public release of the dataset and code would support reproducibility and follow-up work. The paper addresses a real workflow bottleneck and is an engineering contribution with a concrete evaluation on published papers rather than synthetic inputs. However, the central quantitative claim is currently not reproducible from the paper's own tables, and the evaluation protocol rests on a single-annotator ground truth without reported agreement or statistical uncertainty. These issues need to be resolved before the headline numbers can be considered reliable. The authors are also commended for being explicit about the single-annotation limitation and for releasing their system and dataset.","major_comments":[{"comment":"Section 4.1 states that SimAgents 'reduces the overall error rate by 80% compared to CoT and 70% compared to EoT.' No definition of 'overall error rate' reproduces these numbers from Table 2. Summing the average errors per simulation gives CoT = 0.97 + 0.51 + 0.21 = 1.69, EoT = 1.21 + 0.21 + 0.34 = 1.76, and SimAgents = 0.46 + 0.02 + 0.30 = 0.78, corresponding to reductions of 53.8% and 55.7%. Excluding hallucinations gives 67.6% and 66.2%, while the automatic evaluation in Appendix C, which has no hallucination column, gives roughly 84% for both comparisons. The authors should define the metric explicitly and correct either the percentages or the tables. Because this sentence is part of the paper's central quantitative claim, the inconsistency is load-bearing.","section":"Section 4.1, Table 2"},{"comment":"Section 3 states 'Due to time constraints, we only annotated one version of the executable files,' and the Limitations section repeats this limitation. The human evaluation described in Section 3 is said to cover 'as many variants as possible,' but the paper does not report the annotation protocol, the number of annotators, the number of variants considered, or any inter-annotator agreement measure. For papers with ambiguous or multiple valid parameterizations, the precision and recall figures in Table 1 depend on a single annotator's interpretation, so the reported F1 may overstate real-world accuracy. The authors should provide the annotation protocol and agreement statistics, or release the alternative variants.","section":"Section 3, Evaluation; Limitations"},{"comment":"The extraction experiments use GPT-4 on papers that were mostly published before the model's training cutoff, and many of these papers are likely present in GPT-4's training data. This risks overestimating performance on a task that consists of reproducing text-derived parameters. A temporal holdout (for example, papers published after the model's knowledge cutoff) or a comparison on papers the model cannot have memorized would provide a direct test of whether the reported F1 reflects generalization rather than memorization. Without such a test, the headline accuracy numbers remain subject to a contamination concern.","section":"Section 3, Implementation; Table 5"},{"comment":"Section 4.1 says that SimAgents' hallucinated parameters 'are easier to detect and filter,' but Table 2 shows SimAgents has 0.30 hallucination errors per simulation versus 0.21 for CoT, and no detection or filtering mechanism is implemented or evaluated in the paper. As written, this defense is unsupported. The authors should either implement such a filter and measure its effect on the error totals, or temper the claim to acknowledge that hallucination errors remain higher than the CoT baseline.","section":"Section 4.1, Error Analysis"}],"minor_comments":[{"comment":"The phrase 'software-complaint' should be 'software-compliant.'","section":"Section 1, Introduction"},{"comment":"The sentence beginning 'Reduces the overall error rate by 80%...' lacks a grammatical subject and should be merged with the preceding sentence.","section":"Section 4.1, first paragraph"},{"comment":"The dataset size is described as 'more than 40 simulations' in the abstract and Section 3, but the evaluation uses 41 simulations; please make the count consistent throughout.","section":"Section 3, Dataset"},{"comment":"The Qwen3-4B cost is shown as '-'; since the text states that the smaller model has 'no monetary cost,' the table should state this explicitly rather than using a dash.","section":"Appendix D, Table 7"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper is a useful engineering contribution with a public dataset and code release, and I do not see evidence of circularity or fabricated results. The blocking issues are the internally inconsistent 80%/70% error-reduction claim and the thinness of the evaluation protocol (single-annotation ground truth, no inter-annotator agreement, no uncertainty estimates). Both are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: SimAgents looks like a working tool, and the dataset is a real contribution. But the paper's strongest quantitative claim—80% error reduction over CoT and 70% over EoT—does not follow from Table 2. Summing the three error columns gives 1.69 errors per simulation for CoT, 1.76 for EoT, and 0.78 for SimAgents. That is a 54% and 56% reduction, not 80% and 70%. Excluding hallucinations gets you to about 67% and 66%. The automated evaluation in Appendix C, which has no hallucination column, gives roughly 84% for both. No definition in the paper reproduces the stated pair. Since this number is the headline, it is load-bearing and needs a correction, either to the percentages or to the tables.\n\nWhat is genuinely new: the released dataset of 41 cosmology papers annotated with MP-GADGET parameters is the first of its kind, and the design of feeding the Physics Agent only the paper and the Software Agent only the manual is a sensible division of labor that appears to beat both single-agent CoT and symmetric two-agent EoT. Human and automatic evaluations agree on the ranking, which gives me some confidence the effect is real, not just noise. The ablation with Qwen3-4B is a nice touch, and the time/cost numbers are useful.\n\nThe soft spots beyond the metric problem: no error bars or statistical tests on 41 papers, no inter-annotator agreement for the manual annotation, and the authors themselves note they only annotated one executable version per paper. That is an honest limitation, but it does mean the F1 numbers are probably optimistic. GPT-4 training-data contamination is not discussed, though the formatting task is specific enough that contamination is less of a worry than in a pure QA setting. The post-simulation processing section is only lightly evaluated; the 100% execution rate on a subset says little about whether the analysis is scientifically correct.\n\nNone of this kills the paper. The central argument—role specialization helps extraction—is plausible and mostly supported. The paper is sloppy in its headline claim and needs a careful revision of the evaluation section, but the underlying resource and system deserve engagement. I'd send it to peer review with a request to fix the metric definition and add variance or per-paper breakdowns.\n\nWho this is for: cosmology simulation users and anyone building LLM agents for scientific workflows. I'd cite it if I worked on simulation reproducibility.","headline":"Useful system and a genuinely new dataset, but the headline 80%/70% error-reduction claim does not reproduce from the paper's own tables; the real reduction is roughly 54–56%.","tokens_in":10970,"tokens_out":2186,"would_cite":true,"duration_ms":26003,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-agent LLM system converts published cosmology papers into executable simulation files with 98.67% micro-F1.","keywords":["multi-agent LLM","cosmological simulations","parameter extraction","MP-GADGET","large language models","simulation configuration","literature-to-code","scientific reproducibility"],"falsifier":"Rerun the human evaluation with two independent annotators who each produce complete sets of valid MP-GADGET configurations for the same 41 papers; if the annotations disagree substantially, or if SimAgents' micro-F1 against the expanded ground truth falls well below 98.67%, the central accuracy claim would be undermined.","tokens_in":10117,"feed_emoji":"🌌","tokens_out":5165,"duration_ms":56506,"temperature":0.7,"pith_summary":"This paper introduces SimAgents, a multi-agent large language model system that reads a published cosmology paper and produces ready-to-run configuration files for the MP-GADGET simulation code. The system pairs a Physics Agent, which interprets the paper, with a Software Agent, which enforces the simulation software's manual, and has them exchange drafts in an iterative loop. On a new dataset of 41 published simulations, the system reaches a micro-F1 of 98.67%, with precision 97.80% and recall 99.55%, cutting per-simulation parameter errors by roughly 80% relative to a single-agent chain-of-thought baseline. The goal is to compress a task that normally takes a researcher hours to days into about two minutes, and to close the loop from literature reading to initial simulation analysis.","feed_headline":"AI agents read papers, write simulations at 98.67% F1","feed_subtitle":"A two-agent system cuts parameter-extraction errors by 80 percent and finishes in about two minutes.","key_machinery":"The central object is SimAgents, a multi-agent large language model system whose load-bearing mechanism is role separation plus iterative exchange: a Physics Agent sees only the uploaded paper, a Software Agent sees only the simulation software manual, and the two alternate rounds of parameter extraction, coverage checking, and configuration reformatting until the file is physically consistent and software-compliant. The error taxonomy of value errors, type errors, and hallucinations is the yardstick that shows where the specialization helps most, namely by nearly eliminating type errors and halving value errors.","core_discovery":"The central claim is that separating role-specific knowledge into two LLM agents—one for physics reasoning and one for software validation—is enough to turn dense cosmology papers into valid MP-GADGET configurations at near-human accuracy. The paper reports micro-F1 98.67%, precision 97.80%, and recall 99.55% on a manually evaluated dataset of 41 simulations; value errors drop from 0.97 per simulation in the single-agent baseline to 0.46, and type errors from 0.51 to 0.02, while hallucinated parameters remain slightly higher but are said to be easier to flag. An ablation shows that two discussion rounds yield the highest F1, and that switching the backbone model from a commercial large model to a smaller open-weights model lowers F1 to 81.23%. The system also includes an Analysis Code Writer that generates executable Python scripts for power-spectrum plots and density visualizations, with a reported 100% execution rate on the evaluated subset.","pith_inferences":["If the reported accuracy holds across a wider body of literature, literature-to-simulation reproducibility could become routine: every published simulation paper could ship a machine-checked configuration file as a standard artifact.","A testable extension is to measure inter-annotator agreement on the ground truth; if independent annotators disagree about valid parameter choices, the single-annotation evaluation may overstate the system's real-world accuracy.","The same two-agent split (domain reader plus tool-manual validator) could be transferred to other scientific software configuration tasks, and its benefit likely depends on the completeness and unambiguity of the target manual.","A direct stress test would be to run SimAgents on papers that use unusual unit conventions or mixed simulation types, since value errors from unit mismatches are exactly the error class the system only halves rather than eliminates."],"forward_implications":["A published cosmology paper can be converted into an executable MP-GADGET configuration in about two minutes at roughly $0.25 per paper with the commercial backbone, compared with hours to days of manual effort.","The 80% relative reduction in errors means the main remaining risk is subtle value errors such as unit mismatches, which the paper shows can drastically change simulated structure.","The design is modular: swapping the Software Agent's manual should let the same extraction logic target other simulation codes such as GADGET-4, Arepo, GIZMO, or ENZO.","The system extends beyond configuration by generating working preliminary-analysis scripts, so a researcher can move from paper to power spectra and density plots without writing the first analysis code by hand."],"supporting_citations":[{"why":"Supplies MP-GADGET, the simulation software whose configuration syntax and parameters the system must reproduce.","marker":"Feng et al., 2018"},{"why":"Defines the zero-shot chain-of-thought prompting used as the single-agent baseline.","marker":"Kojima et al., 2022"},{"why":"Defines the exchange-of-thought multi-agent baseline that lacks role specialization.","marker":"Yin et al., 2023"},{"why":"Provides GPT-4, the backbone LLM for the main extraction results.","marker":"OpenAI et al., 2023"},{"why":"Provides Qwen3-4B, the smaller backbone model used in the generalizability ablation.","marker":"Yang et al., 2025"},{"why":"Supplies the worked cosmology example whose correct and erroneous MP-GADGET scripts illustrate each error type.","marker":"Zhang et al., 2024"}],"fun_headline_variants":["Two LLM agents turn cosmology papers into sim scripts at 98.67% F1","AI agents read papers, build sims at 98.67% F1 in ~2 minutes","98.67% F1: Multi-agent LLM extracts sim params from papers","LLM agents slash sim-param errors, hit 98.67% F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy rests on a single manually annotated configuration per paper being the right answer; if valid alternative parameter choices were missed, the F1 could overstate real-world performance.","fun_headline_variants_meta":{"raw":{"variants":["Two LLM agents turn cosmology papers into sim scripts at 98.67% F1","AI agents read papers, build sims at 98.67% F1 in ~2 minutes","98.67% F1: Multi-agent LLM extracts sim params from papers","LLM agents slash sim-param errors, hit 98.67% F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00088,"raw_usage":{"total_tokens":3813,"prompt_tokens":962,"completion_tokens":2851,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":2757}},"tokens_in":578,"tokens_out":2851,"duration_ms":24750,"temperature":1.0,"reasoning_tokens":2757,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:07:25.925326+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the human evaluation with two independent annotators who each produce complete sets of valid MP-GADGET configurations for the same 41 papers; if the annotations disagree substantially, or if SimAgents' micro-F1 against the expanded ground truth falls well below 98.67%, the central accuracy claim would be undermined.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the worked cosmology example whose correct and erroneous MP-GADGET scripts illustrate each error type."}],"review_version":1}