{"id":"a21f66ee-420c-4989-9bce-5e52896ec947","arxiv_id":"2606.18543","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Only two of ten advanced AI agents finish a 500-day simulated CEO challenge above the starting cash, and none surpass a hand-tuned rule-based baseline.","lead":"CEO-Bench is a new simulation benchmark that requires AI agents to run a fictional startup for 500 days, balancing pricing, marketing, budgeting and product decisions. On this long-horizon test, most leading language-model agents go bankrupt, while a simple rule-based strategy outperforms them all.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-seed evaluation and best-run selection make the headline failure rates unverified; results may be seed-dependent.","rationale":"The reader identified simulator realism as the weakest assumption, but the more immediate load-bearing risk is internal validity: the entire evaluation uses one simulator seed and best-of-three selection, and the baseline is tuned on that seed. If the seed is unrepresentative, the central empirical claim could be wrong even about the synthetic environment. This is concrete and testable with additional seeds. The realism concern is real but less decisive because the paper itself acknowledges approximation and because the benchmark's immediate claim is about this environment. I therefore keep CONDITIONAL verdict (same as reader) but emphasize a different necessary condition: multi-seed validation. The reader and I partially agree: both note evaluation weaknesses, but we differ on which is load-bearing.","tokens_in":30454,"tokens_out":3288,"duration_ms":36024,"concrete_test":"Re-run the full evaluation for the two best-performing models (Claude Opus 4.8 and GPT-5.5) on at least five independent simulator seeds (e.g., 43, 44, 45, 46, 47) using the same harness, and run the rule-based baseline on each seed without retuning. Report final cash, survival, and bankruptcy counts per seed. If on any seed both models fail to exceed $1M or the baseline, the headline 'only X and Y finish above $1M and all models remain below baseline' is seed-dependent. If results are consistent across seeds, the central claim is substantially strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that current models struggle at long-horizon tasks rests on experiments run entirely with simulator seed 42 (Section 3.1), and the rule-based baseline is tuned on that same seed (Appendix B). All agent runs use that seed; for each model the paper selects the best of three runs for reporting (Section 3.1). This protocol conflates model capability with idiosyncrasies of one random environment realization and with luck across three stochastic runs. The brittleness is visible in Table 3: GPT-5.5 bankrupts in two of three runs and survives only in its best run, yet the abstract and Section 3.2 state that GPT-5.5 'finish[es] above $1M'. Moreover, the abstract says 'all evaluated models remain below the rule-based baseline,' while Table 3 and Figure 2 show Claude Opus 4.8 and GPT-5.5 exceeding that baseline ($27.8M and $21.3M vs $15.76M). This inconsistency suggests result reporting is not robust. If the single seed happens to be unusually favorable or unfavorable to particular strategies, the headline ranking could change. Because the baseline was optimized on this seed, comparing agents to it is not a fair out-of-sample test. Thus the claim that 'most state-of-the-art models struggle' is not yet supported with appropriate uncertainty bounds; it may be an artifact of evaluating one environment draw and selecting best runs.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"CEO-Bench introduces a 500-day simulated startup environment for evaluating LLM agents on long-horizon tasks. Agents manage a fictional company through a Python API, using SQL analytics, pricing, marketing, product development, infrastructure, enterprise sales, and social media to maximize final cash. The paper evaluates ten models (three runs each), reports that most fail to grow the $1M starting balance, identifies Claude Opus 4.8 and GPT-5.5 as the only models finishing above $1M, and compares against a rule-based baseline reaching $15.76M. It also presents ablations on competitor difficulty, time horizon, and agent harness, plus qualitative analyses of agent behavior.","tokens_in":1553,"tokens_out":1663,"duration_ms":69100,"significance":"The benchmark addresses a real gap: existing agent evaluations largely measure short, well-specified tasks, whereas CEO-Bench attempts to measure sustained strategic control under hidden state, delayed feedback, and non-stationarity. The environment is mechanistic, open-ended, and released with trajectories, which are strengths. However, the current experimental evidence is not yet sufficient to support the headline conclusions: all runs use a single environment seed, the rule-based baseline is tuned on that same seed, and results are reported via best-run selection. If the multi-seed evidence and reporting inconsistencies are addressed, the benchmark could be a useful contribution to long-horizon agent evaluation.","major_comments":[{"comment":"All model runs use simulator seed 42, and the rule-based baseline is selected by grid search on the same seed (§3.1 and Appendix B). The paper then reports the best of three runs for each model (§3.1). This protocol does not support claims about model capability: GPT-5.5 bankrupts in 2 of 3 runs with mean survival 333.7±229.7 days, yet the text says it 'finishes above $1M.' Because the three runs are not independent environment realizations, the results provide no estimate of seed-to-seed variation. Please evaluate on multiple independent seeds, report full distributions, and tune the baseline on a separate seed or per-seed so the comparison is out-of-sample.","section":"§3.1, §3.2, Appendix B"},{"comment":"The abstract at the top of the file states 'Only Claude Fable 5, GPT-5.6 Sol, and Claude Opus 4.8 finish above the $1M starting balance, and all evaluated models remain below the rule-based baseline.' The full-text abstract, §3.2, Table 3, and Fig. 2 state instead that Claude Opus 4.8 ($27.8M) and GPT-5.5 ($21.3M) exceed the $15.76M baseline. These statements are mutually incompatible and use different model names. Since the comparison to the rule-based baseline is a central quantitative claim, the manuscript must be reconciled to a single consistent set of numbers and model names.","section":"Abstract vs. §3.2/Table 3"},{"comment":"For surviving models, the paper reports only max final cash and mean survival days; per-run final cash and variance are absent. Claude Opus 4.8 has three surviving runs, but readers cannot tell whether $27.8M is typical or a single favorable outcome. Please report all run-level final cash values, together with mean, median, and interquartile range, for every model. Without this, the qualitative claims about which models 'finish above $1M' rest on selected maxima rather than robust summaries.","section":"Table 3, §3.2"},{"comment":"The simulator mechanics are entirely hand-authored and uncalibrated to empirical data: the customer participation rule (Eq. 5), competitor catch-up model (Eqs. 34–36), and macro Ornstein–Uhlenbeck process (Eq. 37) are plausible but untested abstractions. The abstract and §1 claim agents operate 'in the same environment and the same challenges as a human CEO,' which overstates external validity. Please either temper the realism claims to 'a synthetic environment designed to exercise...' or provide validation or sensitivity analysis (e.g., perturbation of key parameters, calibration to real SaaS/business datasets) showing that the main conclusions are not artifacts of specific parameter choices.","section":"§2.2, Appendix A"}],"minor_comments":[{"comment":"The top-of-file abstract uses model names (Claude Fable 5, GPT-5.6 Sol) that do not appear anywhere in the body; the full-text abstract and body use Claude Opus 4.8 and GPT-5.5. This appears to be a stale or erroneous version and must be fixed.","section":"Abstract"},{"comment":"The typeset name 'CEO-BE N C H' has inconsistent spacing; use 'CEO-Bench' consistently.","section":"Throughout"},{"comment":"The figure shows 'Code / Blog / Trajectory' but no URLs are given in the text. Please provide explicit links to the code, trajectory viewer, and any released data for reproducibility.","section":"Fig. 1, §6.1"},{"comment":"The 'Estimated final cash upper bound' is not a mathematical upper bound: it depends on a chosen configuration and an ad hoc friction factor F=0.49. The appendix does caution that this is an estimate, but the label in Table 3 is misleading. Rename to something like 'Illustrative headroom estimate' and de-emphasize it in the main text.","section":"Appendix D, Table 3"},{"comment":"The plots show means and standard deviations, but the number of runs behind each bar is small (three, with many runs ending early). State the sample sizes and note that early-bankrupt models contribute few observations.","section":"Fig. 12"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and trajectory release are valuable, and the core idea is publishable. The main blockers are the single-seed protocol, best-run selection, and baseline tuning on the same seed, all of which undermine the headline capability claims, plus the inconsistent abstract statements. A revision with multi-seed results, per-run reporting, and a corrected abstract would make the paper suitable for publication. The upper-bound estimate should also be relabeled to avoid overclaiming."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time if you care about agent evaluation beyond single-shot tasks. The environment is the real contribution: a 500-day startup simulation with 26 customer groups, hidden preferences, delayed payoffs, and an adaptive competitor, all mechanistic rather than LLM-judged. That's a genuine step past Vending-Bench and Accounting-Bench. The authors also release trajectories and write a detailed appendix, so the benchmark is reusable. What works: the design principles are sensible, the ablations (competitor difficulty, horizon, harness) give useful information, and the behavioral analysis is suggestive. The fact that the strongest agents write forward-simulators and mine negotiation data is a nice qualitative signal. The soft spots are in the evaluation protocol, not the environment. All runs use seed 42, and the paper reports best-of-three. GPT-5.5 bankrupts in two of three runs; its $21M finish is one lucky trajectory. With a single environment realization, you don't know how much of the ranking is seed-specific. The rule-based baseline is grid-searched on that same seed, so comparing agents to it is not a fair out-of-sample test. And the abstract in the file says all evaluated models stay below the rule-based baseline, but Table 3 and Figure 2 show Claude Opus 4.8 and GPT-5.5 beating it — that's an internal contradiction that needs a fix. None of this kills the benchmark. The core idea is sound, and the paper is honest about many limitations. But the headline 'most models struggle' is only weakly supported: it's true on one seed, with best-run selection, and against a baseline tuned on that seed. A referee should ask for more seeds (at least a handful), error bars or per-run reporting, and a baseline tuned on held-out seeds. The abstract must be corrected. The upper-bound estimate is admittedly rough, which is fine, but the F=0.49 friction factor is arbitrary; worth a caveat. Who is this for? Anyone building or evaluating long-horizon agents. It's a useful resource even if the reported numbers age out. It deserves serious peer review — conditional acceptance with revisions.","headline":"CEO-Bench's environment is a genuine step forward in long-horizon agent evaluation, but the single-seed best-of-three protocol and an abstract/table mismatch weaken the headline claims.","tokens_in":714,"tokens_out":760,"would_cite":true,"duration_ms":54338,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Most state-of-the-art language-model agents go bankrupt when asked to run a simulated startup for 500 days.","keywords":["long-horizon agents","agent benchmark","startup simulation","language-model agents","decision-making under uncertainty","non-stationary environment","delayed feedback","agent evaluation"],"falsifier":"Using the released trajectories, check whether a model run with near-perfect four-week cash forecasts and fast competitor detection still goes bankrupt; if forecasting skill does not track survival, the claimed skill-success link weakens. Alternatively, re-run the simple rule-based baseline under many random seeds other than the single seed used; if its $15.76M result collapses under other seeds, the headline comparison against LLM agents is not stable.","tokens_in":30345,"feed_emoji":"🏢","tokens_out":7477,"duration_ms":68409,"temperature":0.7,"pith_summary":"CEO-Bench is an attempt to measure a capability that short-horizon benchmarks miss: sustaining coherent, adaptive decisions over a long time under noisy and delayed feedback. The paper constructs a 500-day simulation of a subscription-software startup—26 customer groups, 34 tools, a 19-table database, social media, competitors, and macro cycles—and hands control to a language-model agent through a Python terminal. The headline result is that current agents largely fail: most go bankrupt or finish below their $1M starting cash, a simple hand-tuned rule-based policy beats all but the two strongest agents, and the best runs stay far below the estimated achievable ceiling. The paper reads this as evidence that tool-level competence does not yet translate into long-horizon strategic control.","feed_headline":"Rule-based playbook beats almost every AI agent in 500-day startup sim","feed_subtitle":"Only two frontier models finish above the $1M starting cash; the rest go bankrupt or fall short.","key_machinery":"The load-bearing object is the CEO-Bench simulator itself: a mechanistic, stochastic world in which every customer is an individual with a private price-quality tolerance curve, and subscribes only if perceived quality clears a price-dependent minimum requirement. Actions are taken through a composable Python API, and the world responds through coupled equations—reputation spillovers between customer groups, a competitor that raises quality expectations faster when the agent makes broad improvements, a hidden mean-reverting macro cycle, delayed R&D payoffs, and Poisson-distributed acquisition and churn. Because the agent observes only indirect traces, success depends on writing code to query","core_discovery":"The paper's central claim is that evaluating agents on isolated, short-horizon tasks overstates their readiness for real-world work, because the hard part of sustained decision-making is making many interdependent choices whose consequences arrive late, noisily, and through hidden state. CEO-Bench instantiates that challenge by making an agent run NovaMind, a fictional subscription-software company, for 500 simulated days; the agent sees only dashboards, databases, social-media posts, and negotiation histories, and must infer willingness-to-pay, satisfaction, and competitor moves. On this task, most state-of-the-art models cannot survive; the two strongest finish above the starting balance b","pith_inferences":["Inference: the benchmark's conclusions depend on its hand-authored equations and parameters (customer participation, competitor catch-up, macro cycle), which are not calibrated to real market data; a different simulator design could shift which models succeed.","Inference: a strong control experiment would be to put human startup operators on the same interface; if experienced humans beat the best agents, the benchmark captures a real gap, while if they do not, it may reward simulation-specific heuristics.","Inference: since a fixed playbook beats adaptive LLMs, part of the failure may be strategic instability—overreacting to recent noise instead of committing to a plan; testing whether prompt-level 'strategy lock-in' improves outcomes would be a cheap experiment.","Inference: the four skill metrics could be extracted as standalone diagnostic tasks, letting researchers identify which long-horizon skill a model lacks without running the full 500-day simulation."],"forward_implications":["If CEO-Bench measures what it claims, high scores on short-horizon agent benchmarks do not imply ability to steer a long-running organization; readiness must be tested over horizons where costs and benefits compound.","The four measured skills—allocating spend to hidden high-yield channels, forecasting cash four weeks ahead, detecting competitor moves quickly, and forming if-then plans—can serve as progress metrics for long-horizon agent research.","Because even the best agents sit far below the estimated $2.2B ceiling, the benchmark has room to discriminate between current and future models rather than saturating immediately.","A simple rule-based policy that makes no model calls outperforming nearly all LLM agents implies that the bottleneck is not knowledge but sustained, consistent execution of a coherent strategy."],"fun_headline_variants":["Rule-based playbook beats AI agents in 500-day startup sim","Most AI agents fail at 500-day startup challenge","Only two AI models survive 500-day CEO simulation","AI agents lose to simple rules in long-horizon business sim","CEO-Bench: AI agents struggle to run a startup for 500 days"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the hand-built simulator—with its authored customer participation rule, competitor catch-up behavior, and macro cycle, none calibrated to real-world data—is a representative stand-in for the long-horizon decision-making the paper wants to measure; if the mechanics miss the essential difficulties of real sustained management, the benchmark's conclusions may not transfer beyond the simulation.","fun_headline_variants_meta":{"raw":{"variants":["Rule-based playbook beats AI agents in 500-day startup sim","Most AI agents fail at 500-day startup challenge","Only two AI models survive 500-day CEO simulation","AI agents lose to simple rules in long-horizon business sim","CEO-Bench: AI agents struggle to run a startup for 500 days"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000153,"raw_usage":{"total_tokens":1054,"prompt_tokens":763,"completion_tokens":291,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":204}},"tokens_in":507,"tokens_out":291,"duration_ms":3764,"temperature":1.0,"reasoning_tokens":204,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T10:56:33.015264+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Using the released trajectories, check whether a model run with near-perfect four-week cash forecasts and fast competitor detection still goes bankrupt; if forecasting skill does not track survival, the claimed skill-success link weakens. Alternatively, re-run the simple rule-based baseline under many random seeds other than the single seed used; if its $15.76M result collapses under other seeds, the headline comparison against LLM agents is not stable.","supporting_citations":[],"review_version":2}