{"id":"6beec6a7-c3ad-4af8-963c-36262da08530","arxiv_id":"2610.21244","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"System-1 models match fast closed LLMs on static decisions with better-calibrated probabilities at a quarter of the latency, but reasoning LLMs remain most accurate and small System-1 models fail in long-horizon agentic tasks.","lead":"This paper introduces System1-Arena, a benchmark with 92,000 decision questions and 138 simulated environments that tests whether fast, feedforward System-1 models can replace slower generative language models on discrete decision tasks. The results help practitioners decide when to use low-cost, low-latency models versus more expensive reasoning models for applications like content moderation, routing, and agent control.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Closed LLM accuracy may be depressed by the JSON probability output format; the paper provides no control comparison with a simpler prompt.","rationale":"The reader's weakest_assumption identifies prompt fairness as a limitation, noting the paper's fixed single prompt and the acknowledgement that tier-C instructions were written by the authors. My concern is more specific: it targets the JSON probability output format for closed LLMs as a potential source of bias that could affect the central accuracy comparison. I agree with the reader that this is the weakest area, but I drill down on a concrete mechanism (the JSON output format vs. label-only output) and propose a specific test. The reader's verdict of ACCEPT is reasonable, but I think this concern warrants a CONDITIONAL verdict: the paper should demonstrate that the accuracy ranking is robust to at least one alternative prompt format for closed LLMs. If the authors provide this validation or show that the JSON format does not depress accuracy, the concern is resolved and ACCEPT stands. If they cannot, the central claim of System-1 models matching or exceeding fast closed LLMs is not firmly supported.","tokens_in":60991,"tokens_out":9042,"duration_ms":90231,"concrete_test":"Re-evaluate GPT-6 Luna on a random 5,000-question subset of the non-agentic tasks using a label-only prompt (identical in format to the open LLM prompt, Appendix A.2, returning a single label). Obtain the model's confidence as the softmax probability of the first token of the answer label, available through the GitHub Copilot API if logprobs are supported, or through a separate API (e.g., direct endpoint). Compare accuracy to the JSON-prompt result. If the label-only accuracy exceeds 71.2% by more than 1 percentage point, the headline claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that System-1 models match or exceed fast closed LLMs (GPT-6 Luna) on non-agentic accuracy rests on an accuracy difference of 1.4–1.8 percentage points (Table 1). Closed LLMs are evaluated through a fixed prompt that asks for probabilities in JSON (Appendix A.1), which is more complex than the label-selection prompt given to open LLMs (Appendix A.2). The paper acknowledges a fixed prompt per family (Section 4) but does not test whether closed LLM accuracy changes when evaluated via a simpler label-output prompt (even if token probabilities are not available, a label-selection prompt with a post-hoc calibration step could be compared). Because the accuracy gap is small, a shift of a few points from a prompt change could eliminate it. The paper's own paired comparison (Figure 5) shows that GPT-6 Luna (71.2%) outperforms the Decisions API (68.6%), which uses the same underlying model through a structured interface; this suggests the JSON format is not obviously harmful, but a direct controlled experiment with alternative prompts is missing. Without this control, the claim of equivalence or superiority is not robustly established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper introduces System1-Arena, a benchmark with 92k typed questions across 105 static tasks and 138 closed-loop environments, designed to compare System-1 models (which return probabilities directly) with generative LLMs on discrete decisions. The benchmark is constructed from 268 public sources, and the authors evaluate 30 models. The central empirical finding is that the largest System-1 models (Jev, Decision-2.0-Vega-27B) achieve non-agentic accuracy comparable to fast closed LLMs (e.g., GPT-6 Luna at 71.2%) while offering better-calibrated probabilities at roughly one-quarter the latency. Reasoning LLMs (Claude Opus 5.5) remain the most accurate at much higher cost. In closed-loop agentic tasks, large open LLMs succeed most often, while small System-1 models rarely succeed.","tokens_in":61349,"tokens_out":2637,"duration_ms":24282,"significance":"If the main claims hold, this benchmark provides a practical tool for deciding when to use fast, non-generative decision models versus slower generative LLMs. The breadth of tasks (105 tasks, 16 domains) and the inclusion of agentic environments with a real-time mode are methodological contributions. The finding that System-1 models match fast closed LLMs on accuracy while offering better calibration and lower latency is practically relevant for deployment in high-volume, latency-sensitive applications. However, the benchmark's utility depends on whether the model comparisons are robust to evaluation design choices, which are not fully controlled.","major_comments":[{"comment":"Prompt asymmetry between closed and open LLMs. Closed LLMs receive a JSON probability output prompt (Appendix A.1: 'Reply with a single JSON object and nothing else'), while open LLMs receive a simpler label-selection prompt (Appendix A.2: 'Reply with the label of your answer only'). The paper states one fixed prompt per family (Section 4), but this asymmetry could systematically affect accuracy, especially given the small reported gaps (1.4–1.8 pp) between System-1 models and closed LLMs such as GPT-6 Luna. The paper provides no control experiment testing closed LLMs with a simpler label-output prompt (even if token probabilities are not available, a label-selection prompt with post-hoc calibration could be compared). Without this control, the claim that System-1 models 'match or exceed' fast closed LLMs is not robustly established.","section":"Section 4, Appendix A.1 vs A.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Single draw per request and lack of stability analysis. The authors note that models that sample their answers got one draw per request and that run-to-run variation is not measured. This is relevant because the accuracy differences that underpin the main claim are small (1.4–1.8 pp). The paper does not provide confidence intervals or a bootstrap analysis for the overall accuracy rankings in Table 1, nor does it assess how stable the relative ordering of the top models is under repeated evaluation. This weakness is acknowledged but not addressed.","section":"Section 7 (Limitations)","authors_acknowledge":"The models that sample their answers got one draw per request, so run-to-run variation is not measured. 8 Ethical Considerations System1-Arena reuses public datasets and environ- ments under their licenses (Table 4); sources whose terms do not allow redistribution are rebuilt locally by a loader and are not redistributed.","authors_acknowledge_p":0.92},{"comment":"Calibration comparison across different probability elicitation methods. The paper reports that System-1 models have lower Brier scores than fast closed LLMs (e.g., Decision-2.0-Vega-27B 0.343 vs GPT-6 Luna 0.461). However, closed LLMs provide probabilities via verbalized JSON oputput, while open LLMs use answer-token probabilities (Section 4). These different elicitation methods are known to affect calibration (as the paper itself cites Tian et al., 2023; Xiong et al., 2023). The calibration advantage attributed to System-1 models may be partly a methodological artefact. The paper should discuss this caveat when interpreting the Brier score comparison.","section":"Section 3.4, Table 1","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The phrase 'in real-time mode, the world evolves while a model decides' appears twice in the abstract; consider removing one instance.","section":"Abstract","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The row for Jev is labelled 'Jev 1.13'. Clarify whether the '1.13' reffers to the model version; consider removing it from the row label for consistency with other models.\n","section":"Table1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"When describing open LLM probability extraction, the paper says 'GPT-OSS requires reasoning, so we read label probabilities at the start of its final answer.' It may help to explicitly note that gpt-oss-20b and gpt-oss-120b are the two models that use this mode.","section":"Section 4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The axes labels are very small; consider increasing font size for better readability, especially in the scatter plot of accuracy vs latency.","section":"Figure1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The description of 'budget 0' for thinking modes is helpful but could be moved into the main text (Section 5) since it explains the curve origins in Figure 4.\n","section":"Appendix B","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This is the most thorough comparison of System-1 models and generative LLMs on a shared decision interface I have seen. 92k questions, 30 models, 138 environments—the scale alone makes it useful. The real-time mode that advances the environment during inference is a nice touch. The authors are transparent about their limitations: label noise, contamination, run-to-run variation, and the fact that 33% of instructions are author-written (tier C). They check that agentic harnesses are not too easy via uniform-random baselines. The code and data appear to be public. All of this is good practice.\n\nThe core result—that the largest System-1 models match fast closed LLMs (e.g., GPT-6 Luna) on non-agentic accuracy with better calibration at one-quarter the latency—holds up under the data presented. The accuracy gaps are small (1.4–1.8 pp), but the pattern is consistent across domains. The paired comparison (Decisions API vs. GPT-6 Luna) shows that the same underlying model loses 2.6 pp when accessed through the structured interface, which actually suggests the JSON prompt is not the cause of Luna's lower score relative to Jev or Decision-2.0-Vega-27B.\n\nThat said, the stress-test concern is real: the paper does not control for prompt format. Closed LLMs get a JSON instruction; open and System-1 models get different prompts. The paper acknowledges a fixed prompt per family, but a simple ablation (e.g., asking closed LLMs to output a label instead of JSON, then post-hoc calibrating) would have addressed it directly. Given the small accuracy margins, a prompt change could shift rankings. This is a minor weakness, not a fatal one—the authors are forthright about the limitation, and the paired comparison partially mitigates the worry.\n\nThe benchmark is squarely aimed at practitioners who need to decide whether to route a decision to a fast, calibrated model or pay for generative reasoning. Researchers studying System-1 architectures or calibration will also find value. I would cite this as a reference dataset for decision-focused evaluation.\n\nVerdict: deserves a serious referee. I would accept it with a minor comment asking for a control experiment on prompt format, but the paper is already strong enough without it.","headline":"A careful, large-scale benchmark that gives practical answers about where System-1 models are ready—and where they are not. The main finding is robust despite a minor confound in prompting.","tokens_in":61824,"tokens_out":1549,"duration_ms":18071},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"System1-Arena introduces a benchmark to determine which discrete decisions can be left to fast System-1 models and which still require generative reasoning.","keywords":["System-1 models","System-2 models","discrete decisions","benchmark","probability calibration","agentic tasks","cost-latency tradeoff","LLM evaluation"],"falsifier":"If a System-1 model matched or exceeded Claude Opus 5.5 on non-agentic tasks after per-task prompt optimization, the claim that reasoning LLMs are necessary for best accuracy would be weakened.","tokens_in":62522,"feed_emoji":"⚡","tokens_out":3455,"duration_ms":37188,"temperature":0.7,"pith_summary":"The paper aims to answer which discrete decisions can be handled by System-1 models—models that return probabilities in a single forward pass—versus generative LLMs. It introduces System1-Arena, a benchmark with 92k questions across 105 tasks and 138 closed-loop environments, and evaluates 30 models. The largest System-1 models (Jev, Decision-2.0-Vega-27B) match or exceed fast closed LLMs (GPT-6 Luna) on non-agentic accuracy, with better-calibrated probabilities at a quarter of the latency. Reasoning LLMs (Claude Opus 5.5) achieve the highest accuracy at about 70x the cost. In closed-loop agentic tasks, large open LLMs succeed most often; small System-1 models rarely succeed.","feed_headline":"Fast decision models match slow LLMs on most static tasks","feed_subtitle":"System1-Arena shows Jev and Decision-2.0 match GPT-6 Luna accuracy with better calibration at quarter the latency.","key_machinery":"System1-Arena's decision interface, which represents every example and environment step as a request containing a state and typed questions (boolean, choice, score). This standardizes input across all model families and enables probability scoring. A real-time protocol advances the environment during model inference to measure the value of fast decisions, and a method for obtaining decision probabilities from open autoregressive and diffusion LLMs through answer-token log-probabilities is provided.","core_discovery":"The paper establishes that the largest System-1 models can replace generative LLMs for many static decision tasks without sacrificing accuracy, while providing better probability calibration at much lower cost. It demonstrates this by comparing models on a shared decision interface across diverse tasks, finding that Decision-2.0-Vega-27B and Jev achieve 73.0% and 72.6% accuracy respectively, comparable to GPT-6 Luna's 71.2%, with Brier scores of 0.343 and 0.349 versus 0.461. The paper also discovers that step accuracy does not predict closed-loop success, and that forecasting remains a difficult domain for System-1 models, with no model exceeding 61.6% accuracy.","pith_inferences":["The benchmark could be extended to multi-modal decisions (image, audio) to test System-1 models in richer domains, potentially revealing where fast, single-pass processing fails further.","The fixed prompt per family may disadvantage some models; per-task prompt tuning could change rankings, especially for instruction-sensitive tasks where wording strongly influences performance.","The poor calibration of answer-token probabilities in open LLMs (e.g., Gemma-4-31B with Brier score 0.474 despite 73.9% accuracy) suggests that using these probabilities for decision thresholds may be unreliable, a gap that System-1 models fill."],"forward_implications":["System-1 models can be deployed for high-volume, low-latency decisions like phishing detection, safety moderation, and routing, reducing cost and latency without sacrificing accuracy.","The benchmark provides a standard for comparing System-1 and System-2 models, enabling informed routing decisions between them based on task type.","The finding that step accuracy does not correlate with closed-loop success suggests that evaluating agents requires full environment interaction, not static step accuracy.","Forecasting tasks remain challenging for System-1 models, indicating that some domains still require the broader knowledge or reasoning of larger LLMs."],"supporting_citations":[{"why":"JevBench, the prior System-1 benchmark with a few hundred synthetic decisions, whose narrow coverage the paper addresses.","marker":"Benchmark Heaven, 2026"},{"why":"ReflexBench, another prior System-1 benchmark with limited scope and no agentic tasks.","marker":"Brida AI, 2026"},{"why":"Provides the Brier score used to evaluate probability quality throughout the paper.","marker":"Brier, 1950"},{"why":"Defines the System 1 / System 2 distinction that frames the paper's central question.","marker":"Kahneman, 2011"},{"why":"HELM benchmark that reports calibration and efficiency, providing context for System1-Arena's design.","marker":"Liang et al., 2022"}],"fun_headline_variants":["System-1 models match GPT-6 Luna on static tasks at quarter latency","Decision-2.0-Vega-27B rivals fast LLMs with better calibration, lower cost","System1-Arena shows fast decision models can replace LLMs on many tasks","Large System-1 models deliver LLM accuracy on static decisions for 70x less"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The benchmark uses a fixed, single prompt per model family, which may not be equally fair to all models, and 33% of task instructions were written by the authors, inheriting label noise from sources and potentially affecting model rankings.","fun_headline_variants_meta":{"raw":{"variants":["System-1 models match GPT-6 Luna on static tasks at quarter latency","Decision-2.0-Vega-27B rivals fast LLMs with better calibration, lower cost","System1-Arena shows fast decision models can replace LLMs on many tasks","Large System-1 models deliver LLM accuracy on static decisions for 70x less"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":9.6e-05,"raw_usage":{"cost":9.59469e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":9.59469e-05,"upstream_inference_prompt_cost":4.273794e-05,"upstream_inference_completions_cost":5.320896e-05},"total_tokens":1543,"prompt_tokens":951,"completion_tokens":592,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":501}},"tokens_in":951,"tokens_out":592,"cost_source":"openrouter_usage","duration_ms":10619,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":501,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-16T15:57:14.720292+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"If a System-1 model matched or exceeded Claude Opus 5.5 on non-agentic tasks after per-task prompt optimization, the claim that reasoning LLMs are necessary for best accuracy would be weakened.","supporting_citations":[{"cited_title":"Overview of the TREC 2019 deep learning track","cited_arxiv_id":"2003.07820","evidence_quote":"JevBench, the prior System-1 benchmark with a few hundred synthetic decisions, whose narrow coverage the paper addresses."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}