CausalGame: Benchmarking Causal Thinking of LLM Agents in Games

Bo Han; Chenxi Liu; Jialin Li; Junchi Yu; Kun Zhang; Philip Torr; Xiangchen Song; Yongqiang Chen; Zhenhao Chen; Zijian Li

REVIEW 3 major objections 6 minor 1 cited by

No frontier LLM agent shows reliable causal thinking on interactive science-style games that hide selection bias, measurement error, and confounders.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 20:18 UTC pith:FYSYJ27G

load-bearing objection Solid interactive benchmark that actually embeds selection bias, measurement error, and hidden confounders; the multi-model shortfall is real, but the leap from game scores to “no reliable causal thinking” is still a proxy claim. the 3 major comments →

arxiv 2607.04293 v1 pith:FYSYJ27G submitted 2026-07-05 cs.CL cs.AIcs.LGstat.ML

CausalGame: Benchmarking Causal Thinking of LLM Agents in Games

Zhenhao Chen , Yongqiang Chen , Chenxi Liu , Junchi Yu , Xiangchen Song , Zijian Li , Jialin Li , Philip Torr

show 2 more authors

Bo Han Kun Zhang

This is my paper

classification cs.CL cs.AIcs.LGstat.ML

keywords causal thinkingLLM agentsAI Scientistselection biashidden confoundersmeasurement errorinteractive benchmarkstructural causal model

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

CausalGame turns scientific discovery into interactive drone-design games whose outcomes are governed by fixed but hidden structural causal models. Agents must design limited experiments, read censored or noisy outcomes, and submit both a final design and an explanation. Across 14 scenarios built around selection bias, measurement error, and hidden confounders, 30 frontier LLM agents never approach the analytically optimal survival rates (about 78–85%), and only a few percent of sessions earn credit for identifying the true causal mechanism. Survival often comes from trial-and-error without mechanistic understanding, and extra compute or stronger agent scaffolding raises scores without fixing that gap. The benchmark argues that causal thinking for AI Scientist agents should be judged by interventions against a fixed hidden mechanism, not by correlational fit or the agent’s own narrative.

Core claim

On CausalGame’s 14 interactive scenarios, none of 30 evaluated LLM agents demonstrates reliable causal thinking: the best agentic survival is 68.0% against family optima of roughly 78–85%, only about 5–7% of sessions receive credit on the causal-reasoning rubrics, and high survival is routinely decoupled from identifying the underlying mechanism under selection bias, measurement error, and hidden confounders.

What carries the argument

CausalGame: a two-stage interactive game (limited exploration budget, then one final fleet evaluation) whose environment is a structural causal model (SCM) that injects selection bias (survived drones only), measurement noise, and hidden confounders (e.g., antenna signal emission or EMI), with survival plus a rubric-scored explanation report as the dual score.

Load-bearing premise

That shortfalls on these fixed SCM drone games with tight exploration budgets and LLM-judged explanation rubrics mainly measure missing causal thinking rather than task framing, exploration limits, or judge noise on the causal criteria.

What would settle it

An LLM agent that, under the same hardened API and budgets, consistently clears win thresholds near the SCM optima while earning non-zero credit on the core causal-reasoning criteria (true mechanism, trap avoidance, and mechanistic depth) across Antenna Trap, Deployment Zone Trap, and Weather Noise families—not merely high survival by trial-and-error.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Progress claims for AI Scientist agents should report interventional outcomes against a fixed hidden SCM, not only narrative quality or correlational analysis of observed variables.
Existing general capability benchmarks are weak proxies for this skill: CausalGame ranks only weakly with them, so high scores elsewhere do not imply causal thinking under bias and confounding.
More reasoning tokens or stronger agent harnesses can raise survival without raising causal-understanding scores, so scaffolding alone is not a substitute for the missing capability.
Interactive discovery benchmarks need leak-hardening and external victory criteria, because agents will mine side channels and declare false success.
Training or evaluation loops can use CausalGame’s procedurally fixed scenarios to demand joint gains in survival and causal-rubric credit rather than search heuristics alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the same gap holds outside games, automated hypothesis and experiment agents will keep rediscovering spurious associations unless they are forced to plan interventions that identify hidden common causes.
The decoupling of survival from explanation suggests reward designs that optimize only outcome metrics will reinforce correlational shortcuts rather than mechanism recovery.
Configuration-path failures (component lock-in, drift away from already-found optima) point to a behavioral exploration problem, not only a reporting problem, that pure text rubrics might understate.
A natural next test is whether agents trained or fine-tuned on CausalGame-style SCMs transfer to other active causal-discovery settings with different surface stories but the same pitfall structure.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid interactive benchmark that actually embeds selection bias, measurement error, and hidden confounders; the multi-model shortfall is real, but the leap from game scores to “no reliable causal thinking” is still a proxy claim.

read the letter

The useful part of this paper is the testbed, not the slogan. CausalGame is a multi-turn drone-design game whose engine is a fixed SCM, with 14 scenarios that deliberately inject selection bias (survive-only observations), measurement noise, and hidden confounders (Antenna Trap, Deployment Zone / Farr-style EMI, weather noise). Agents get a limited exploration budget, then a final design plus a short report scored on causal reasoning, experimental design, reflection, and data use. That combination is a real gap relative to workflow suites and clean data-analysis benches; Table 1 is fair about it.

What they did well: analytical optima from the SCM, checked empirically within a few points; win thresholds set below optima; 30 models, prompting vs agentic (and a small OpenCode slice); non-LLM baselines; Spearman links to other benches; multi-judge ICC; configuration-path analysis that does not rely on the judge; and honest reporting of API leaks and false victory claims, then a hardened setup. Survival tops out around 68% against ~78–85% optima, CR credit is rare, and high survival often comes without mechanism. That pattern is documented carefully enough that I trust the numbers as numbers.

The soft spot is attribution, not fabrication. The stress-test is right that “none demonstrates reliable causal thinking” leans on a proxy: fixed budget, censored feedback, and CR rubrics that are 87–92% zero with only moderate ICC on the causal axes. Non-LLM baselines overlapping the weak agents, and OpenCode still far from optimum, are consistent with hard search under bias as well as with pure causal failure. The paper itself shows survival and understanding can decouple, which is interesting, but it also means the headline wording is stronger than the identification. Mode comparisons are multi-factor, not clean ablations. Those are caveats, not reasons to dismiss the work.

Who it is for: people building AI-scientist agents, causal-ML evals, and interactive discovery benches. Math is standard SCM, citations are appropriate, data design is careful. I would bring it to reading group, cite the benchmark and the empirical gap, and send it to referees. Accept-shaped contribution with the proxy claim dialed back in revision.

Referee Report

3 major / 6 minor

Summary. The paper introduces CausalGame, an interactive benchmark that casts scientific causal discovery as drone-design games driven by fixed structural causal models (SCMs). Agents allocate defense across components under a limited exploration budget (200 drones, ≤10 deployments), then submit a final design and a natural-language report. Fourteen scenarios instantiate selection bias, measurement error, and hidden confounders via two main families (Antenna Trap, Deployment Zone Trap) plus Weather Noise. Evaluation combines Stage-2 fleet survival against analytically derived optima (∼78–85%) and scenario-specific win thresholds with a four-dimension LLM-as-judge rubric (causal reasoning, experimental design, reflection, data usage). Across 30 frontier models in prompting and agentic modes (3 trials each), the best agentic survival is 68.0%, causal-reasoning rubric credit is rare (about 5–7% of sessions), survival is largely decoupled from mechanistic explanation, and scores correlate only weakly with existing capability benchmarks. The authors also document non-LLM baselines, scaffolding ablations, configuration-path failure modes, and evaluation-hardening after API leakage and false-victory claims.

Significance. If the empirical picture holds, CausalGame fills a clear gap: existing AI-Scientist and interactive-discovery benchmarks largely omit observational pitfalls (selection bias, measurement error, hidden confounders) under which correlational strategies are systematically misleading. The combination of a fixed hidden SCM, interventional survival outcomes, and explanation rubrics is a useful evaluation design, and the paper ships substantial supporting apparatus—analytical optima verified within ±2–3 pp, non-LLM baselines, multi-judge ICC, judge-independent configuration-path analysis, Spearman comparisons to external benchmarks, and transparent reporting of hacking behaviors with a patched suite. That package makes the work a concrete, falsifiable testbed for causal thinking in agentic LLMs rather than another narrative-only science benchmark.

major comments (3)

The headline claim that “none demonstrates reliable causal thinking” (Abstract; §4.2 Observation 1) attributes the survival gap (best 68.0% vs family optima ~78–85%) primarily to missing causal reasoning. That attribution is load-bearing but only partly isolated from exploration constraints. Stage 1 allows only 200 drones and ≤10 deployments with survival-censored (and sometimes noisy) feedback (§3.1). Non-LLM baselines already reach 49–57.5% and overlap the lower agent range (Fig. 8b; Appendix G), and even OpenCode leaves a large gap to optimum (Fig. 8a). The paper never reports an oracle or mechanism-aware policy run under the same budget and observation model to show that the analytical optimum is discoverable—not merely evaluable—within the protocol. Without that (or an equivalent budget-sufficiency argument), shortfalls can be read as hard search under censored feedback rather than
Causal-reasoning rubric scores (CR1–CR3) are the second pillar of the central claim (Abstract; Fig. 6; §4.4), yet they are extremely skewed (87–92% zeros) with only moderate inter-rater ICC(2,3) ~0.61–0.64 (Fig. 11; Appendix). The authors correctly note that skew depresses ICC, and they add configuration-path analysis as a judge-independent check (Fig. 13). Still, the abstract’s “5–7% of sessions receive credits on the causal-reasoning rubrics” is presented as direct evidence of failed causal thinking. Given moderate CR agreement and the fact that high-survival tiers still sit far from saturated CR polygons (Fig. 6), the main text should (i) state ICC and zero-mass for CR up front when citing those percentages, (ii) report sensitivity of the 5–7% figure to judge choice / partial-credit rules, and (iii) avoid equating low CR credit with absence of causal competence without that caveat.
Prompting vs Agentic comparisons are used to discuss scaffolding effects (Observation 5; Table 16; Table 5), but the two modes differ simultaneously in tool access, mandatory ReAct formatting, exploration guards, and tool limits. The paper acknowledges this in Appendix A and notes the comparison is not a controlled ablation, yet the main narrative still draws mode-level conclusions (e.g., agentic sensitivity to selection bias in Table 3; failure-mode ring charts in Fig. 12). Either run a controlled factor ablation (one change at a time) or reframe all mode contrasts as multi-factor system comparisons and remove causal language about “agentic scaffolding” as a single mechanism.

minor comments (6)

Figure 1 and several result figures use model names that do not always match Table 11 / the experimental list (e.g., Claude-Opus-4.7 vs 4.5, GPT-5.5 family). Harmonize naming across figures, tables, and text.
Win thresholds and family optima are summarized in Table 8 and Fig. 25; a single main-text table listing per-scenario threshold, analytical optimum, and empirical optimum would make Observation 1 easier to audit without the appendix.
Table 1 is helpful; briefly clarify in the caption how “Observational pitfalls” differs from NewtonBench / BoxingGym / Causal AI Scientist so the uniqueness claim is self-contained.
The Weather Noise family uses a 55% threshold vs 75% elsewhere (Table 6). Flag this more visibly when averaging survival across all 14 scenarios so readers do not misread cross-family means.
Appendix A limitations are appropriate; move a short paragraph on proxy scope (simplified SCM games vs open-ended science) into the main Conclusions so the abstract’s generality claim is balanced in the body.
Minor polish: “none explicitly incorporates” (Abstract) vs “none explicitly incorporate” agreement; consistent hyphenation of “causal-reasoning”; ensure Eq. (1) variable notation matches later SCM tables.

Circularity Check

0 steps flagged

No circular derivation: CausalGame is an empirical benchmark with fixed SCMs, analytically derived optima, and externally measured agent outcomes.

full rationale

The paper does not claim a first-principles derivation whose target is forced by its inputs. Family optima are obtained by maximizing expected Stage-2 survival under the SCM (Eq. 2 / Appendix C.4) and verified on 1,000-drone fleets within ±2–3 pp; win thresholds are set below those optima by design, not fitted to agent scores. Survival and rubric scores are interventional/external measurements against a fixed hidden SCM and a task-report ground truth, not renamings of fitted parameters. Self-citations (e.g., Liu et al. 2024; Chen et al. 2026) appear in related-work and future-work discussion and are not load-bearing uniqueness theorems that force the main empirical claim. Defining both the trap scenarios and the evaluation criteria is standard benchmark construction, not self-definitional circularity: agent performance remains free to vary and is reported against non-LLM baselines and multiple execution modes. Concerns about proxy validity (budget, exploration, judge ICC on CR) are correctness/construct-validity issues, not circular reductions of predictions to inputs. Score 0 with no circular steps.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 3 invented entities

Load-bearing content is mostly engineered evaluation design: SCMs, budgets, thresholds, and rubric weights chosen by the authors to instantiate known causal pitfalls. No physical free parameters are fitted to nature; free choices are protocol hyperparameters. Background axioms are standard SCM semantics and the domain claim that these three observational pitfalls are central to scientific discovery. Invented entities are the benchmark, scenario families, and scoring rubrics themselves.

free parameters (4)

exploration_budget = 200 drones, ≤10 calls
Stage 1 allows 200 drones and up to 10 deployment calls; these limits shape how much interventional evidence agents can gather and thus affect measured capability.
win_thresholds = mostly 75%; weather 55%
Scenario-specific victory thresholds (typically ~5–8 pp below family optima; 55% for Weather Noise) are author-chosen cutoffs that define ‘win’ and influence win-rate summaries.
rubric_point_weights = 11+2+2+1
Causal reasoning (11), experimental design (2), reflection (2), data usage (1) weights determine overall rubric scores and failure-mode taxonomy.
SCM_noise_and_detection_parameters = family-specific (Appendix C.3)
Family templates fix detection/EMI/noise coefficients (e.g., weather noise σ, EMI reduction rates, detection base rates) that set difficulty and optimal survival; chosen to create traps rather than estimated from external data.

axioms (4)

domain assumption Structural causal models with interventions correctly formalize the target scientific skill of distinguishing causation from correlation under bias and confounding.
Stated throughout §§1–3 and Table 2; evaluation interprets failure against the SCM as failure of causal thinking.
domain assumption Observing only survivors (and related selection mechanisms) induces the intended spurious correlations analogous to Berkson/Sackett-type selection bias.
Antenna Trap design in §3.2 and Fig. 4; used to claim agents fall into correlational traps.
ad hoc to paper LLM-as-judge rubric scores, averaged over three judges, are a valid measure of causal explanation quality despite skewed CR distributions.
§4.4 and Appendix E; authors report ICC and add configuration-path checks, but CR criteria remain mostly zero.
standard math Standard probability and SCM composition rules for computing optimal survival rates from structural equations.
Appendix C.4 optimal survival definition r★_s = max_a E[Y_s(a,U)].

invented entities (3)

CausalGame benchmark (drone designer + SCM engine) no independent evidence
purpose: Provide a controlled interactive testbed for causal thinking of AI scientist agents.
Core contribution; not claimed as a natural object outside the evaluation suite.
Antenna Trap and Deployment Zone Trap scenario families no independent evidence
purpose: Instantiate selection bias and hidden-confounder traps with known optimal designs.
Synthetic environments inspired by historical cases but engineered for the benchmark.
Four-dimension causal report rubric (CR/ED/RQ/DU) no independent evidence
purpose: Score whether explanations recover mechanisms rather than only high survival.
Author-defined scoring instrument; reliability partially validated via multi-judge ICC.

pith-pipeline@v1.1.0-grok45 · 59321 in / 3610 out tokens · 40519 ms · 2026-07-11T20:18:04.741145+00:00 · methodology

0 comments

read the original abstract

Building AI Scientist agents with Large Language Models (LLMs) has recently attracted growing attention. Since scientific discovery fundamentally relies on uncovering causal relationships from observations, the capability of causal thinking, i.e., distinguishing causation from correlation and recognizing hidden biases, is essential to LLM agents. Although a number of benchmarks exist for AI Scientists, none explicitly incorporate challenges from selection bias, measurement error, and hidden confounders that widely exist in real-world scientific discovery. To this end, we present CausalGame, a benchmark that evaluates the causal thinking capabilities of LLM agents through interactive games. CausalGame asks LLM agents to actively design experimental protocols, collect observation data, and derive a final solution with an explanation report. To emulate realistic scientific discovery challenges, we design 14 scenarios that incorporate selection bias, measurement error, and hidden confounders. Across 30 LLM agents, none demonstrates reliable causal thinking: the best model reaches only 68.0% survival against analytical optima of 78-85%, and merely 5-7% of sessions receive credits on the causal-reasoning rubrics. CausalGame provides a scalable and controlled testbed for evaluating the causal thinking of AI Scientist agents.

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

CausalDS: Benchmarking Causal Reasoning in Data-Science Agents
cs.AI 2026-07 accept novelty 7.0

CausalDS generates SCM-grounded scenes with free-form stories and noisy observations to jointly score causal reasoning, coding, uncertainty, and abstention in data-science agents.

Reference graph

Works this paper leans on

26 extracted references · 6 linked inside Pith · cited by 1 Pith paper

[1]

(Cited on pages 4 and 28) Artificial Analysis

URLhttps://openreview.net/forum?id=EO8mTLqDuT. (Cited on pages 4 and 28) Artificial Analysis. Artificial analysis long context reasoning (AA-LCR) benchmark.https://arti ficialanalysis.ai/articles/announcing-aa-lcr, August 2025. Accessed: 2026-05-29. (Cited on page 13) V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan.𝜏2-bench: Evaluating conversational...

Pith/arXiv arXiv 2025
[2]

(Cited on pages 2, 7 and 8) 18 CausalGame: Benchmarking Causal Thinking of LLM Agents in Games P. Feng, Z. Lv, J. Ye, X. Wang, X. Huo, J. Yu, W. Xu, W. Zhang, L. Bai, C. He, et al. Earth-agent: Unlocking the full landscape of earth observation with agents.arXiv preprint arXiv:2509.23141,

arXiv
[3]

Gandhi, M

(Cited on page 27) K. Gandhi, M. Y. Li, L. Goodyear, A. Bhatia, Y. Li, A. Bhaskar, M. Zaman, and N. Goodman. Boxinggym: Benchmarking progress in automated experimental design and model discovery. InWorkshop on Scaling Environments for Agents, 2025. URLhttps://openreview.net/forum?id=TgobzsU0 3X. (Cited on pages 4 and 28) J. Geng, H. Chen, D. Arumugam, and...

Pith/arXiv arXiv doi:10.1038/s41586-025-09422-z 2025
[4]

Jansen, M.-A

(Cited on page 13) P. Jansen, M.-A. Côté, T. Khot, E. Bransom, B. D. Mishra, B. P. Majumder, O. Tafjord, and P. Clark. Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URLhttps://openreview.n...

Pith/arXiv arXiv 2024
[5]

Researchbench: Benchmarking llms in scientific discovery via inspiration-based task decomposition.arXiv preprint arXiv:2503.21248, 2025

(Cited on page 27) Y.Liu, Z.Yang, T.Xie, J.Ni, B.Gao, Y.Li, S.Tang, W.Ouyang, E.Cambria, andD.Zhou. Researchbench: Benchmarking llms in scientific discovery via inspiration-based task decomposition.arXiv preprint arXiv:2503.21248, 2025. (Cited on pages 2, 4 and 28) C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha. The ai scientist: Towards full...

Pith/arXiv arXiv doi:10.1037/0033-2909.86.2.420 2025
[6]

doi: 10.1093/0195155270.001.0001

ISBN 9780195155273. doi: 10.1093/0195155270.001.0001. URLhttps://doi.org/10 .1093/0195155270.001.0001. (Cited on page 28) Y. Yamada, R. T. Lange, C. Lu, S. Hu, C. Lu, J. Foerster, J. Clune, and D. Ha. The ai scientist- v2: Workshop-level automated scientific discovery via agentic tree search.arXiv preprint arXiv:2504.08066, 2025. (Cited on pages 2, 4 and ...

Pith/arXiv arXiv doi:10.1093/0195155270.001.0001 2025
[7]

(Cited on page 29) C. Yang, J. Lu, H. Wan, J. Yu, and F. Qin. From what to why: A multi-agent system for evidence-based chemical reaction condition reasoning. InThe Fourteenth International Conference on Learning Representations, 2026. (Cited on page 27) Z. Yang, W. Liu, B. Gao, T. Xie, Y. Li, W. Ouyang, S. Poria, E. Cambria, and D. Zhou. MOOSE-chem: Larg...

2026
[8]

(Cited on page 9) W. Yao, Y. Sun, A. Ho, C. Sun, and K. Zhang. Learning temporally causal latent processes from general temporal data.International Conference on Learning Representations 2022, 2021. (Cited on page 29) W. Yao, G. Chen, and K. Zhang. Temporally disentangled representation learning.Conference and Workshop on Neural Information Processing Sys...

arXiv 2022
[9]

stealth mode

(Cited on page 28) K. Zhang, S. Xie, I. Ng, and Y. Zheng. Causal representation learning from multiple distributions: A general setting.International Conference on Machine Learning 2024, 2024. (Cited on page 29) 25 CausalGame: Benchmarking Causal Thinking of LLM Agents in Games S. Zhang, S. Yang, T. Xie, X. Xue, Z. Hu, R. Li, W. Qu, Z. Yin, T. Fu, D. Hu, ...

Pith/arXiv arXiv 2024
[14]

**TEST**: Design experiments to validate your theories ## GAME FLOW
[15]

**STAGE 1 (Exploration)**: Use ‘client.deploy_drone()‘ to test hypotheses - You have {{total_drones}} drones for experimentation - Each deployment returns: survival status, hit_count, environment data
[16]

black box

**STAGE 2 (Validation)**: Call ‘client.submit_final_design(design)‘ - Runs {{stage2_fleet_size}} simulations with your final design - **WARNING**: You can only submit ONCE - this is irreversible! ## VICTORY CONDITION - **Survival Rate** >= {{victory_threshold}}% ## AVAILABLE METHODS - ‘client.get_history()‘ - Get all flight history - ‘client.get_all_envir...
[17]

**OBSERVE**: Look for patterns in survival, hit_count, and environmental conditions
[18]

**DISCOVER**: Query for hidden measurements (weather, atmospheric, etc.)
[19]

**ANALYZE**: Check how different factors relate to outcomes
[20]

**HYPOTHESIZE**: Form theories about what drives survival
[21]

**TEST**: Design experiments to validate your theories ## QUESTIONS TO CONSIDER - Why do some drones get hit more often? Is it random or systematic? - Does higher DEF always improve survival? Are there trade-offs? - What environmental factors matter? Are there hidden variables? - How do different components affect overall performance? ## GAME FLOW
[22]

**STAGE 1 (Exploration)**: Deploy drones to test hypotheses - You have {{total_drones}} drones for experimentation - Each deployment returns: survival status, hit_count, environment data
[23]

**STAGE 2 (Validation)**: Submit your final design - Runs {{stage2_fleet_size}} simulations with your final design - **WARNING**: You can only submit ONCE - this is irreversible! - **CRITICAL**: You MUST submit before the mission ends! ## VICTORY CONDITION - **Survival Rate** >= {{victory_threshold}}% - Optimize your design to maximize drone survival ## A...
[24]

Return to step 1 with new information ReAct Instruction Injection.Before each turn, the following instruction is injected into the agent’s context to enforce reasoning: Listing 3|ReAct Instruction (Injected Each Turn) [IMPORTANT: ReAct Format] Before calling any tool, you MUST first explain your reasoning:
[25]

What did you observe from previous results?
[26]

What is your hypothesis?
[27]

Why are you taking this action? Output your THOUGHT first, then call the tool. Post-Deployment Analysis Prompt.After each deploy_drone call returns results, an additional analysis prompt is appended to encourage systematic reasoning: Listing 4|Analysis Prompt (After Deployment Results) [ANALYZE THIS RESULT]
[28]

What is the survival rate? Does it match your expectation?
[29]

What does this tell you about the design parameters?
[30]

this could be improved

What should you test next to validate or refine your hypothesis? Safety Guards.The Agentic mode implements several safety mechanisms: • ExplorationGuard: Agentsmustcall deploy_droneatleastoncebefore submit_final_design is allowed. This prevents premature submissions without data collection. • Tool Iteration Limit: Maximum of 5-10 tool calls per turn (conf...

[1] [1]

(Cited on pages 4 and 28) Artificial Analysis

URLhttps://openreview.net/forum?id=EO8mTLqDuT. (Cited on pages 4 and 28) Artificial Analysis. Artificial analysis long context reasoning (AA-LCR) benchmark.https://arti ficialanalysis.ai/articles/announcing-aa-lcr, August 2025. Accessed: 2026-05-29. (Cited on page 13) V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan.𝜏2-bench: Evaluating conversational...

Pith/arXiv arXiv 2025

[2] [2]

(Cited on pages 2, 7 and 8) 18 CausalGame: Benchmarking Causal Thinking of LLM Agents in Games P. Feng, Z. Lv, J. Ye, X. Wang, X. Huo, J. Yu, W. Xu, W. Zhang, L. Bai, C. He, et al. Earth-agent: Unlocking the full landscape of earth observation with agents.arXiv preprint arXiv:2509.23141,

arXiv

[3] [3]

Gandhi, M

(Cited on page 27) K. Gandhi, M. Y. Li, L. Goodyear, A. Bhatia, Y. Li, A. Bhaskar, M. Zaman, and N. Goodman. Boxinggym: Benchmarking progress in automated experimental design and model discovery. InWorkshop on Scaling Environments for Agents, 2025. URLhttps://openreview.net/forum?id=TgobzsU0 3X. (Cited on pages 4 and 28) J. Geng, H. Chen, D. Arumugam, and...

Pith/arXiv arXiv doi:10.1038/s41586-025-09422-z 2025

[4] [4]

Jansen, M.-A

(Cited on page 13) P. Jansen, M.-A. Côté, T. Khot, E. Bransom, B. D. Mishra, B. P. Majumder, O. Tafjord, and P. Clark. Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URLhttps://openreview.n...

Pith/arXiv arXiv 2024

[5] [5]

Researchbench: Benchmarking llms in scientific discovery via inspiration-based task decomposition.arXiv preprint arXiv:2503.21248, 2025

(Cited on page 27) Y.Liu, Z.Yang, T.Xie, J.Ni, B.Gao, Y.Li, S.Tang, W.Ouyang, E.Cambria, andD.Zhou. Researchbench: Benchmarking llms in scientific discovery via inspiration-based task decomposition.arXiv preprint arXiv:2503.21248, 2025. (Cited on pages 2, 4 and 28) C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha. The ai scientist: Towards full...

Pith/arXiv arXiv doi:10.1037/0033-2909.86.2.420 2025

[6] [6]

doi: 10.1093/0195155270.001.0001

ISBN 9780195155273. doi: 10.1093/0195155270.001.0001. URLhttps://doi.org/10 .1093/0195155270.001.0001. (Cited on page 28) Y. Yamada, R. T. Lange, C. Lu, S. Hu, C. Lu, J. Foerster, J. Clune, and D. Ha. The ai scientist- v2: Workshop-level automated scientific discovery via agentic tree search.arXiv preprint arXiv:2504.08066, 2025. (Cited on pages 2, 4 and ...

Pith/arXiv arXiv doi:10.1093/0195155270.001.0001 2025

[7] [7]

(Cited on page 29) C. Yang, J. Lu, H. Wan, J. Yu, and F. Qin. From what to why: A multi-agent system for evidence-based chemical reaction condition reasoning. InThe Fourteenth International Conference on Learning Representations, 2026. (Cited on page 27) Z. Yang, W. Liu, B. Gao, T. Xie, Y. Li, W. Ouyang, S. Poria, E. Cambria, and D. Zhou. MOOSE-chem: Larg...

2026

[8] [8]

(Cited on page 9) W. Yao, Y. Sun, A. Ho, C. Sun, and K. Zhang. Learning temporally causal latent processes from general temporal data.International Conference on Learning Representations 2022, 2021. (Cited on page 29) W. Yao, G. Chen, and K. Zhang. Temporally disentangled representation learning.Conference and Workshop on Neural Information Processing Sys...

arXiv 2022

[9] [9]

stealth mode

(Cited on page 28) K. Zhang, S. Xie, I. Ng, and Y. Zheng. Causal representation learning from multiple distributions: A general setting.International Conference on Machine Learning 2024, 2024. (Cited on page 29) 25 CausalGame: Benchmarking Causal Thinking of LLM Agents in Games S. Zhang, S. Yang, T. Xie, X. Xue, Z. Hu, R. Li, W. Qu, Z. Yin, T. Fu, D. Hu, ...

Pith/arXiv arXiv 2024

[10] [14]

**TEST**: Design experiments to validate your theories ## GAME FLOW

[11] [15]

**STAGE 1 (Exploration)**: Use ‘client.deploy_drone()‘ to test hypotheses - You have {{total_drones}} drones for experimentation - Each deployment returns: survival status, hit_count, environment data

[12] [16]

black box

**STAGE 2 (Validation)**: Call ‘client.submit_final_design(design)‘ - Runs {{stage2_fleet_size}} simulations with your final design - **WARNING**: You can only submit ONCE - this is irreversible! ## VICTORY CONDITION - **Survival Rate** >= {{victory_threshold}}% ## AVAILABLE METHODS - ‘client.get_history()‘ - Get all flight history - ‘client.get_all_envir...

[13] [17]

**OBSERVE**: Look for patterns in survival, hit_count, and environmental conditions

[14] [18]

**DISCOVER**: Query for hidden measurements (weather, atmospheric, etc.)

[15] [19]

**ANALYZE**: Check how different factors relate to outcomes

[16] [20]

**HYPOTHESIZE**: Form theories about what drives survival

[17] [21]

**TEST**: Design experiments to validate your theories ## QUESTIONS TO CONSIDER - Why do some drones get hit more often? Is it random or systematic? - Does higher DEF always improve survival? Are there trade-offs? - What environmental factors matter? Are there hidden variables? - How do different components affect overall performance? ## GAME FLOW

[18] [22]

**STAGE 1 (Exploration)**: Deploy drones to test hypotheses - You have {{total_drones}} drones for experimentation - Each deployment returns: survival status, hit_count, environment data

[19] [23]

**STAGE 2 (Validation)**: Submit your final design - Runs {{stage2_fleet_size}} simulations with your final design - **WARNING**: You can only submit ONCE - this is irreversible! - **CRITICAL**: You MUST submit before the mission ends! ## VICTORY CONDITION - **Survival Rate** >= {{victory_threshold}}% - Optimize your design to maximize drone survival ## A...

[20] [24]

Return to step 1 with new information ReAct Instruction Injection.Before each turn, the following instruction is injected into the agent’s context to enforce reasoning: Listing 3|ReAct Instruction (Injected Each Turn) [IMPORTANT: ReAct Format] Before calling any tool, you MUST first explain your reasoning:

[21] [25]

What did you observe from previous results?

[22] [26]

What is your hypothesis?

[23] [27]

Why are you taking this action? Output your THOUGHT first, then call the tool. Post-Deployment Analysis Prompt.After each deploy_drone call returns results, an additional analysis prompt is appended to encourage systematic reasoning: Listing 4|Analysis Prompt (After Deployment Results) [ANALYZE THIS RESULT]

[24] [28]

What is the survival rate? Does it match your expectation?

[25] [29]

What does this tell you about the design parameters?

[26] [30]

this could be improved

What should you test next to validate or refine your hypothesis? Safety Guards.The Agentic mode implements several safety mechanisms: • ExplorationGuard: Agentsmustcall deploy_droneatleastoncebefore submit_final_design is allowed. This prevents premature submissions without data collection. • Tool Iteration Limit: Maximum of 5-10 tool calls per turn (conf...