{"id":"9f35adeb-44dc-4062-aed9-ea75d6f422ae","arxiv_id":"2608.06871","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CEDAR pairs an LLM editor with an LLM judge inside tree search to rewrite Python simulations of complex systems toward natural-language goals, and beats a numeric optimizer on one record-fitting benchmark.","lead":"This paper introduces CEDAR, a system that uses two AI assistants, an editor and a judge, to repeatedly rewrite computer models of complex systems until they satisfy a goal described in plain English. The authors show it improving a 200-year world model and fitting a noisy population curve more accurately than a standard numerical optimizer in one comparison.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The vague-goal result is confounded by depth-dependent score inflation: the Judge caps scores at 10 + 2.5*depth while selection adds gamma*depth, so deeper nodes are favored regardless of goal quality; the reported 'best' system may just be the deepest.","rationale":"I read the paper as making three capability claims: vague-goal optimization, record fitting, and interpretability. The record-fitting experiment is externally benchmarked and is the strongest support; the abstract-goal experiment is the novel differentiator and the one that depends on the LLM Judge as the fitness signal. The reader identified the general risk that the LLM Judge is a subjective, possibly circular evaluator. My stress test sharpens that into a concrete, internal scoring artifact: the Judge's score upper bound grows linearly with tree depth, and the selection rule adds another depth bonus, so the algorithm's objective is partially 'maximize depth' rather than 'maximize goal satisfaction.' This is not merely a philosophical concern about LLM judgment; it is a testable property of the prompt and the UCT formula. The theoretical section's assertion that the reward is bounded is also inaccurate across depths, since the bound depends on depth. The transcripts corroborate that the LLM is explicitly aware of the depth-dependent cap and uses it in its self-assessments. The constraint violation in the edits is additional evidence that the system is not reliably enforcing the user's stated goal. Together these do not overturn the record-fitting result, but they mean the paper's central novelty is currently supported only by a scoring signal that is confounded. A re-run with external metrics and depth-invariant scoring would settle whether the concern lands; hence CONDITIONAL remains the appropriate verdict, with the condition being that the abstract-goal experiment is re-evaluated without the depth-inflation artifact.","tokens_in":33607,"tokens_out":5383,"duration_ms":60807,"concrete_test":"Re-run the World Dynamics abstract-goal experiment with an independent, external evaluator: from each node's execution record compute the three objective metrics (final population, resource remaining, pollution level) and score nodes on a fixed absolute scale. Also rerun with the Judge prompt's depth cap removed and with gamma = 0. If the node selected as best under original CEDAR scoring is not the node with best external metrics, or if removing depth-dependent terms materially changes the selected node's external metrics, the vague-goal claim rests on score inflation. Additionally, check the returned system's helper coefficients against the stated 'no change >50%' constraint; violations would show the goal was not satisfied as written.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The load-bearing premise for the abstract-goal experiment is that the Judge's score S_v measures goal satisfaction on a comparable scale across nodes. This premise fails internally. The Judge prompt fixes a depth-dependent cap ('score <= 10.0 + 2.5 * depth'), while node selection adds an explicit depth bonus: SCORE(v) = S_v + gamma*DEPTH(v) with gamma = 2. So the same behavior earns a higher total score at greater depth even before the bonus; S_v is not uniformly bounded across the tree, contradicting the Theoretical Connection's claim that S_u constitutes a bounded, non-adversarial reward. Algorithm 1 returns the node with highest S_u, so the 'best' abstract-goal system may simply be the deepest one. In the provided transcripts, scores rise monotonically along the path (6.75 -> 9.25 -> 11.75 -> 13.25) and the Editor itself cites the depth allowance ('10.0 + 2.5*4 = 20.0'), showing the models are exploiting the allowance. The record-fitting result is immune because it uses external L1/DTW, but the paper's flagship 'vague natural-language goal' capability is not yet evidenced independently of this scoring artifact. A further sign the stated goal was not met literally: the Editor responses repeatedly violate the goal's own 'change no coefficient by more than 50%' constraint (e.g., NRMM reduced by 75-95%, POLCM by 90-98%).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CEDAR, an MCTS framework in which an LLM Editor proposes variants of a complex system represented in a restricted Python subset and an LLM Judge assigns scores and textual analyses against a natural-language goal. The authors formalize node selection as a generalized UCT with progressive widening and a depth bonus, report a qualitative demonstration on the World Dynamics model, a quantitative record-fitting comparison against Optuna using external L1/DTW metrics, interpretability analyses, and an MCTS-versus-linear ablation. The paper concludes with explicit limitations, including reliance on LLM judgment for abstract goals and the absence of tight statistical claims.","tokens_in":33893,"tokens_out":5932,"duration_ms":59951,"significance":"Conditional on the record-fitting result, CEDAR's ability to discover dynamics without predefined formula structure is a meaningful contribution, and the use of external ground-truth metrics in a 3500-step setting is a real strength. The abstract-goal demonstration, however, is not yet load-bearing evidence because its fitness signal is confounded by depth-dependent scoring and by the shared LLM Judge, so the paper's flagship capability claim is not established. The manuscript is transparent about several of these weaknesses, which is commendable, but the empirical core needs revision before the claims can be accepted.","major_comments":[{"comment":"The vague-goal experiment is confounded by the depth-dependent score cap. The Judge prompt sets 'Maximum possible score is score <= 10.0 + 2.5 * depth', while node selection uses SCORE(v) = S_v + ... + gamma * DEPTH(v) with gamma = 2, so both the raw Judge score and the acquisition score grow with depth independent of goal quality. Along the reported best path, scores increase 6.75, 9.25, 11.75, 13.25, and Editor Response 6 explicitly cites the allowance ('10.0 + 2.5*4 = 20.0'). Algorithm 1 returns the node with maximum S_u, so the 'best' system may be deep rather than goal-satisfying. The Theoretical Connection's claim that S_u constitutes a bounded, non-adversarial reward is inconsistent with a depth-dependent cap. Please deconfound by removing the depth cap, normalizing scores, comparing nodes at equal depth, or providing external or human evaluation.","section":"Fitting an Abstract Goal; Appendix Method Details; Theoretical Connection"},{"comment":"The literal 50% coefficient constraint is violated in the reported run. The goal text states 'only change the coefficients in the helper. Do not change any coefficient by more than 50%', but the Editor responses along the reported path announce NRMM reductions of 75-95%, POLCM reductions of 90-98%, and CIM increases of up to 15x. The paper does not report constraint-adherence statistics or discuss these violations. Since the Judge is the sole evaluator and appears not to enforce the stated constraint, the demonstration does not show that CEDAR satisfies the user-specified goal as written.","section":"Fitting an Abstract Goal; Interpretability transcripts"},{"comment":"The headline record-fitting claim rests on single runs. Table 2 gives one L1/DTW value per CEDAR backend and one Optuna full-formulae run; the appendix adds a second Optuna run but no CEDAR seed variability or significance test. The paper's own conclusion says 'We report trends rather than tight statistical claims,' but the abstract and introduction present the Table 2 comparison as the main quantitative evidence. Please report multiple seeds for both CEDAR and Optuna, with distributions or confidence intervals, before claiming that CEDAR outperforms Optuna with full formulae.","section":"Table 2 and Quantitative Studies: Fitting a Concrete Record"},{"comment":"The ablation supporting the claim that 'MCTS leads to better performance' appears to rely on a single run or very few runs, with no statistical quantification. The section asserts that MCTS is crucial for performance gains, but Figure 8 shows only node scores and trajectories without seed variability or a test. The comparison should also control for the number of LLM calls or total compute, since CEDAR and linear search may differ in budget. This component claim needs additional support.","section":"Why Use MCTS with LLMs; Figure 8"}],"minor_comments":[{"comment":"The word 'diveristy' should be 'diversity'.","section":"Conclusion"},{"comment":"Equation (1) uses 'cu+1' in the denominator where the main text defines the expansion count of node v as cv, and the beta term is not defined in the main text.","section":"Theoretical Connection: Full Details, Eq. (1)"},{"comment":"The text says the Sakoe-Chiba band with window size w=250 restricts the warping path to within 205 time steps; this number appears inconsistent with w=250 and should be corrected or clarified.","section":"Appendix: Dynamic Time Warping Distance"},{"comment":"The Related Work section says the method 'scales to 60 variables and 4000 steps', while Experimental Details report 20 systems with 20 to 69 integrated variables; these numbers should be reconciled or stated as approximate.","section":"Related Work and Experimental Details"},{"comment":"The appendix overview contains unresolved placeholder section numbers such as 'Section .' and should be populated with the correct references.","section":"Appendix Overview"},{"comment":"No code or data availability statement is provided; releasing the prompts, the harness, and the dataset conversions would materially aid reproducibility.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the flagship vague-goal result is a scoring artifact: the depth-dependent cap, combined with gamma*depth selection and the LLM Judge as the only evaluator, makes the reported 'best' system hard to interpret. The authors are honest about circularity and about reporting trends, but these concessions are not reflected in the experimental design. If the authors can deconfound the scoring and add multi-seed evidence for the record-fitting claim, the paper may be publishable; in the current form I would not accept the central claims as stated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The part worth your attention is the record-fitting experiment. CEDAR, starting from a bare-minimum skeleton with no formulae, beats Optuna that is given the full ground-truth formulae (L1 2.22 vs 3.71, DTW 433 vs 477). That is a real, externally benchmarked result, and the paper is honest about favoring the baseline: Optuna gets 100 trials and full structure, CEDAR gets neither. The representation of system dynamics as a restricted Python subset with Euler integration looks sensible, and scaling to 69 integrated variables and 4000 steps is a genuine step beyond the small 4-variable systems in prior LLM dynamics work. The writing is clear, and the authors disclose the main circularity in the conclusion: the LLM Judge scores candidates and shares a model class with the Editor. Good.\n\nThe soft spots are real, and one is load-bearing. The vague-goal World Dynamics result, which the paper treats as a flagship capability, is confounded by the scoring function itself. The Judge prompt caps scores at 10 + 2.5*depth, and node selection adds gamma*depth with gamma=2. So a node at depth 4 can be scored up to 20, while a shallow node can never exceed 10 plus its own depth bonus. The reported scores along the best path rise monotonically (6.75, 9.25, 11.75, 13.25), and the Editor response explicitly refers to the depth constraint ('10.0 + 2.5*4 = 20.0'). That means the 'best' system may simply be the deepest one, not the one that best satisfies the natural-language goal. This is not just the circularity the authors admit—it is an internal inconsistency with the theoretical claim that S_u is a bounded reward. The abstract-goal experiment needs an independent evaluator or at least a fixed score scale across depths. Also, the stated 'do not change any coefficient by more than 50%' constraint is violated repeatedly in the transcripts (NRMM reduced 75-95%, POLCM 90-98%), so the constraint is not enforced. The record-fitting results are single-run in Table 2; the appendix adds an extra Optuna run but no CEDAR seed variability or statistical test, consistent with the paper's own caveat that it reports trends rather than tight claims.\n\nFor the right audience—people working on LLM-guided optimization, system dynamics, or ALife—this paper is worth a serious look, but the abstract-goal section should be treated as suggestive, not demonstrative. It deserves peer review because the record-fitting result is plausible and the method is described in enough detail to reproduce. But I would ask for seeds, a fixed score scale, and enforcement checks on the coefficient constraint before accepting the broader claims.","headline":"The record-fitting result is genuinely interesting and honestly presented, but the abstract-goal demonstration is confounded by a depth-dependent scoring cap, so the flagship claim is not yet established.","tokens_in":747,"tokens_out":1001,"would_cite":false,"duration_ms":31639,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CEDAR claims an LLM Judge–Editor loop inside Monte Carlo tree search can discover complex systems that satisfy natural-language behavioral goals, and can fit a target record better than a baseline that is given the full ground-truth…","keywords":["complex systems","system dynamics","Monte Carlo Tree Search","large language models","evolutionary computation","emergent behavior","goal-directed optimization","record fitting"],"falsifier":"Run CEDAR on the World Dynamics goal but replace the Judge's score with an external fixed multi-objective metric (for example, a weighted sum of final population, remaining resources, and pollution level), then check whether the system selected as best under the LLM Judge also optimizes that external metric; if the Judge's scores and the external metric diverge, the abstract-goal claim is not supported.","tokens_in":33363,"feed_emoji":"🌳","tokens_out":5170,"duration_ms":46923,"temperature":0.7,"pith_summary":"The paper is trying to establish that goal-directed design of complex systems—nonlinear feedback models of populations, resources, pollution, and the like—can be automated by a search loop in which a large language model judges how well a candidate system's simulated behavior meets a natural-language goal, and a second LLM edits the system's code to improve that score. The intended payoff is that a researcher can say \"grow the population sustainably and reduce pollution\" and get back a working system, without hand-writing equations in a specialized modeling language. If true, this would lower the barrier to system-dynamics modeling, let vague human goals drive structural discovery, and make the search process interpretable through the LLMs' written analyses. The paper's strongest quantitative claim is that CEDAR, starting from a bare skeleton and given no formulas, fits a stochastic population record with lower L1 and DTW error than an optimizer that is handed the full ground-truth formulas.","feed_headline":"LLM tree search beats full-formula fitting with no formulas","feed_subtitle":"A judge-editor loop inside MCTS fits stochastic population records without knowing the equations.","key_machinery":"The load-bearing object is the search tree over system programs, formalized as an MCTS variant whose node-selection score is $S_v + \\phi(c_v, c_p, \\tau)$, where $\\phi$ combines progressive widening ($\\alpha\\sqrt{\\ln(c_p+1)/(c_v+1)}$ capped at expansion threshold $\\tau$) with a depth bonus $\\gamma\\cdot\\mathrm{DEPTH}(v)$. The LLM Editor acts as a stochastic transition kernel $P_\\theta(P_u \\mid P_v, A_v, s, G)$ over programs, and the LLM Judge acts as a learned value function producing $(A_u, S_u)$ with a bounded, noisy reward $S_u$; crashed or NaN-valued runs receive low scores so the tree backtracks. The representation layer—a restricted Python subset with four marked sections (initialization, intermediate computations, derivatives, Euler integration), domain wrappers such as graph(), delay(), smth1(), and inline edit markers—is what lets the LLMs make structural edits rather than only tuning coefficients.","core_discovery":"The central claim is that CEDAR turns complex-system discovery into Monte Carlo tree search over executable programs: each node is a system written in a restricted, runnable Python subset with Euler-method integration; an LLM Editor proposes modified systems (adding variables, rewiring feedbacks, changing equations), the edited system is executed, and an LLM Judge returns a bounded numerical score plus a textual analysis. Because the Judge scores behavior against the user's natural-language goal, the search can optimize abstract objectives such as \"balance population, resources, and environment\" without a predefined scalar fitness function. The paper reports that on the World Dynamics system, the search improves all three targeted variables together, and that in the record-fitting experiment CEDAR without predefined formulations (L1 2.22, DTW 433.13 for GPT-5.1) outperforms Optuna with the full formula skeleton (L1 3.71, DTW 477.52).","pith_inferences":["A testable extension is to keep the Judge and Editor but swap the executable representation for agent-based models or cellular automata; the same two-operator loop should transfer if the representation enforces the same structural edit markers.","Because the Judge and Editor share a model class, the abstract-goal results may partly measure the LLM's internal notion of \"sustainable balance\" rather than an external ground truth; a held-out external multi-objective metric would separate these.","The record-fitting result suggests that the hard part of scientific model discovery is proposing new functional structure, not tuning coefficients; an extension would compare CEDAR against symbolic regression or equation-discovery methods on the same records."],"forward_implications":["A user can specify a goal in natural language and receive a working system-dynamics model, removing the need to write equations in DYNAMO- or STELLA-style languages.","Starting from a bare skeleton, the method can reconstruct the behavior of a stochastic ground-truth system more accurately than an optimizer given the full ground-truth formulas, implying that structure discovery rather than parameter fitting drives the gain.","Keeping a tree rather than a single trajectory preserves several distinct high-scoring systems, enabling sensitivity analysis across different trade-off emphases.","Every edit and evaluation carries a textual rationale, so the search's path from initial system to final system is inspectable, even though the texts are not verified causal explanations.","MCTS with the same LLM operators scores higher than a linear (single-chain) search, so the branching exploration contributes directly to performance."],"supporting_citations":[{"why":"Supplies the World Dynamics system used for the abstract-goal experiment and the largest system in the dataset.","marker":"(Forrester, 1971)"},{"why":"Supplies the 19 biological systems from STELLA that form the rest of the evaluation dataset.","marker":"(Hannon and Ruth, 2014)"},{"why":"Defines the STELLA modeling language whose workflow limitations motivate the Python-based representation.","marker":"(Richmond, 1985)"},{"why":"Provides the UCT/MCTS foundation that CEDAR's node-selection score generalizes.","marker":"(Kocsis and Szepesvári, 2006)"},{"why":"Supplies the progressive widening principle used in the generalized UCT score.","marker":"(Chaslot et al., 2008)"},{"why":"Establishes LLMs as variation operators in evolutionary search, grounding the LLM Editor design.","marker":"(Lehman et al., 2023)"},{"why":"Defines the Optuna black-box optimizer used as the quantitative baseline in record fitting.","marker":"(Akiba et al., 2019)"},{"why":"Provides the DTW metric used alongside L1 distance to measure trajectory fit.","marker":"(Sakoe and Chiba, 1978)"}],"fun_headline_variants":["Agent tree search hits goals without predefined formulas","LLM judge-editor MCTS beats full-formula fitting on records","Goal-driven LLM search designs complex systems beyond formulas","CEDAR: program-space search optimizes system behavior by goals","No formulas, just goals: LLM tree search finds better fits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM Judge's bounded score is a faithful, non-adversarial measure of how well a system satisfies the user's natural-language goal, especially in the abstract-goal experiment where no external metric is used.","fun_headline_variants_meta":{"raw":{"variants":["Agent tree search hits goals without predefined formulas","LLM judge-editor MCTS beats full-formula fitting on records","Goal-driven LLM search designs complex systems beyond formulas","CEDAR: program-space search optimizes system behavior by goals","No formulas, just goals: LLM tree search finds better fits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000334,"raw_usage":{"total_tokens":1896,"prompt_tokens":1027,"completion_tokens":869,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":786}},"tokens_in":643,"tokens_out":869,"duration_ms":8409,"temperature":1.0,"reasoning_tokens":786,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:11:22.754779+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CEDAR on the World Dynamics goal but replace the Judge's score with an external fixed multi-objective metric (for example, a weighted sum of final population, remaining resources, and pollution level), then check whether the system selected as best under the LLM Judge also optimizes that external metric; if the Judge's scores and the external metric diverge, the abstract-goal claim is not supported.","supporting_citations":[],"review_version":1}