{"arxiv_id":"2610.21244","title":"System1-Arena: Which Decisions Can Be Left to System 1?","abstract":"Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human. System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. Which decisions can System-1 models handle, and which require generative reasoning? We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed.","primary_cat":"cs.CL","cross_cats":["cs.AI","cs.LG"],"submitted_at":"2026-10-15T04:40:47+00:00","latest_version":1,"source":"arxiv","source_label":"arXiv","source_url":null,"has_pdf":true,"has_source":false,"verdict":"ACCEPT","confidence":"HIGH","one_line_summary":"System-1 models match fast closed LLMs on static decisions with better-calibrated probabilities at a quarter of the latency, but reasoning LLMs remain most accurate and small System-1 models fail in long-horizon agentic tasks.","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","upvotes":0,"downvotes":0,"pith_score":0,"inbound_count":7,"_links":{"html":"https://pith.science/paper/2610.21244","verdict":"/api/v1/verdicts/2610.21244"}}