Computation and Language · arXiv:2610.21244 v1 · submitted 15 Oct 2026 · 31 references
System1-Arena: Which Decisions Can Be Left to System 1?
The paper says
“Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human.”
more of the abstract
System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. Which decisions can System-1 models handle, and which require generative reasoning? We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 16 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
The passes disagreed. Reader: holds up. Skeptic: holds up only with conditions. Referee: major revision.
A careful, large-scale benchmark that gives practical answers about where System-1 models are ready—and where they are not. The main finding is robust despite a minor confound in prompting. This line is wrong
3 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 31 references say.
The paper says, at Section 4, Appendix A.1 vs A.2
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
Prompt asymmetry between closed and open LLMs. Closed LLMs receive a JSON probability output prompt (Appendix A.1: 'Reply with a single JSON object and nothing else'), while open LLMs receive a simpler label-selection prompt (Appendix A.2: 'Reply with the label of your answer only').
more of objection 1
The paper states one fixed prompt per family (Section 4), but this asymmetry could systematically affect accuracy, especially given the small reported gaps (1.4–1.8 pp) between System-1 models and closed LLMs such as GPT-6 Luna. The paper provides no control experiment testing closed LLMs with a simpler label-output prompt (even if token probabilities are not available, a label-selection prompt with post-hoc calibration could be compared). Without this control, the claim that System-1 models 'match or exceed' fast closed LLMs is not robustly established.
The paper says, at Section 7 (Limitations)
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.92):
“The models that sample their answers got one draw per request, so run-to-run variation is not measured. 8 Ethical Considerations System1-Arena reuses public datasets and environ- ments under their licenses (Table 4); sources whose terms do not allow redistribution are rebuilt locally by a loader and are not redistributed.”
Pith says: Objection 2 of 3 · major
Single draw per request and lack of stability analysis. The authors note that models that sample their answers got one draw per request and that run-to-run variation is not measured. This is relevant because the accuracy differences that underpin the main claim are small (1.4–1.8 pp).
more of objection 2
The paper does not provide confidence intervals or a bootstrap analysis for the overall accuracy rankings in Table 1, nor does it assess how stable the relative ordering of the top models is under repeated evaluation. This weakness is acknowledged but not addressed.
The paper says, at Section 3.4, Table 1
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
Calibration comparison across different probability elicitation methods. The paper reports that System-1 models have lower Brier scores than fast closed LLMs (e.g., Decision-2.0-Vega-27B 0.343 vs GPT-6 Luna 0.461). However, closed LLMs provide probabilities via verbalized JSON oputput, while open LLMs use answer-token probabilities (Section 4).
more of objection 3
These different elicitation methods are known to affect calibration (as the paper itself cites Tian et al., 2023; Xiong et al., 2023). The calibration advantage attributed to System-1 models may be partly a methodological artefact. The paper should discuss this caveat when interpreting the Brier score comparison.
5 minor objections
The paper says, at Abstract
Pith says: minor objection 1 of 5
The phrase 'in real-time mode, the world evolves while a model decides' appears twice in the abstract; consider removing one instance.
The paper says, at Table1
Pith says: minor objection 2 of 5
The row for Jev is labelled 'Jev 1.13'. Clarify whether the '1.13' reffers to the model version; consider removing it from the row label for consistency with other models.
show 3 more
The paper says, at Section 4
Pith says: minor objection 3 of 5
When describing open LLM probability extraction, the paper says 'GPT-OSS requires reasoning, so we read label probabilities at the start of its final answer.' It may help to explicitly note that gpt-oss-20b and gpt-oss-120b are the two models that use this mode.
The paper says, at Figure1
Pith says: minor objection 4 of 5
The axes labels are very small; consider increasing font size for better readability, especially in the scatter plot of accuracy vs latency.
The paper says, at Appendix B
Pith says: minor objection 5 of 5
The description of 'budget 0' for thinking modes is helpful but could be moved into the main text (Section 5) since it explains the curve origins in Figure 4.
What would settle it
Re-evaluate GPT-6 Luna on a random 5,000-question subset of the non-agentic tasks using a label-only prompt (identical in format to the open LLM prompt, Appendix A.2, returning a single label). Obtain the model's confidence as the softmax probability of the first token of the answer label, available through the GitHub Copilot API if logprobs are supported, or through a separate API (e.g., direct endpoint). Compare accuracy to the JSON-prompt result. If the label-only accuracy exceeds 71.2% by more than 1 percentage point, the headline claim weakens.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 16 Oct 2026
- Reader holds up
- Skeptic partly agrees: says it holds up only with conditions Closed LLM accuracy may be depressed by the JSON probability output format; the paper provides no control comparison with a simpler prompt.
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This is the most thorough comparison of System-1 models and generative LLMs on a shared decision interface I have seen. 92k questions, 30 models, 138 environments—the scale alone makes it useful. The real-time mode that advances the environment during inference is a nice touch. The authors are transparent about their limitations: label noise, contamination, run-to-run variation, and the fact that 33% of instructions are author-written (tier C). They check that agentic harnesses are not too easy via uniform-random baselines. The code and data appear to be public. All of this is good practice.
The core result—that the largest System-1 models match fast closed LLMs (e.g., GPT-6 Luna) on non-agentic accuracy with better calibration at one-quarter the latency—holds up under the data presented. The accuracy gaps are small (1.4–1.8 pp), but the pattern is consistent across domains. The paired comparison (Decisions API vs. GPT-6 Luna) shows that the same underlying model loses 2.6 pp when accessed through the structured interface, which actually suggests the JSON prompt is not the cause of Luna's lower score relative to Jev or Decision-2.0-Vega-27B.
That said, the stress-test concern is real: the paper does not control for prompt format. Closed LLMs get a JSON instruction; open and System-1 models get different prompts. The paper acknowledges a fixed prompt per family, but a simple ablation (e.g., asking closed LLMs to output a label instead of JSON, then post-hoc calibrating) would have addressed it directly. Given the small accuracy margins, a prompt change could shift rankings. This is a minor weakness, not a fatal one—the authors are forthright about the limitation, and the paired comparison partially mitigates the worry.
The benchmark is squarely aimed at practitioners who need to decide whether to route a decision to a fast, calibrated model or pay for generative reasoning. Researchers studying System-1 architectures or calibration will also find value. I would cite this as a reference dataset for decision-focused evaluation.
Verdict: deserves a serious referee. I would accept it with a minor comment asking for a control experiment on prompt format, but the paper is already strong enough without it.
The strongest objection
The central claim that System-1 models match or exceed fast closed LLMs (GPT-6 Luna) on non-agentic accuracy rests on an accuracy difference of 1.4–1.8 percentage points (Table 1). Closed LLMs are evaluated through a fixed prompt that asks for probabilities in JSON (Appendix A.1), which is more complex than the label-selection prompt given to open LLMs (Appendix A.2). The paper acknowledges a fixed prompt per family (Section 4) but does not test whether closed LLM accuracy changes when evaluated via a simpler label-output prompt (even if token probabilities are not available, a label-selection prompt with a post-hoc calibration step could be compared). Because the accuracy gap is small, a shift of a few points from a prompt change could eliminate it. The paper's own paired comparison (Figure 5) shows that GPT-6 Luna (71.2%) outperforms the Decisions API (68.6%), which uses the same underlying model through a structured interface; this suggests the JSON format is not obviously harmful, but a direct controlled experiment with alternative prompts is missing. Without this control, the claim of equivalence or superiority is not robustly established.
The reader's weakest_assumption identifies prompt fairness as a limitation, noting the paper's fixed single prompt and the acknowledgement that tier-C instructions were written by the authors. My concern is more specific: it targets the JSON probability output format for closed LLMs as a potential source of bias that could affect the central accuracy comparison. I agree with the reader that this is the weakest area, but I drill down on a concrete mechanism (the JSON output format vs. label-only output) and propose a specific test. The reader's verdict of ACCEPT is reasonable, but I think this concern warrants a CONDITIONAL verdict: the paper should demonstrate that the accuracy ranking is robust to at least one alternative prompt format for closed LLMs. If the authors provide this validation or show that the JSON format does not depress accuracy, the concern is resolved and ACCEPT stands. If they cannot, the central claim of System-1 models matching or exceeding fast closed LLMs is not firmly supported.
The referee's report
If the main claims hold, this benchmark provides a practical tool for deciding when to use fast, non-generative decision models versus slower generative LLMs. The breadth of tasks (105 tasks, 16 domains) and the inclusion of agentic environments with a real-time mode are methodological contributions. The finding that System-1 models match fast closed LLMs on accuracy while offering better calibration and lower latency is practically relevant for deployment in high-volume, latency-sensitive applications. However, the benchmark's utility depends on whether the model comparisons are robust to evaluation design choices, which are not fully controlled.
Pith's reading of the paper (a machine's summary, not the authors')
The paper aims to answer which discrete decisions can be handled by System-1 models—models that return probabilities in a single forward pass—versus generative LLMs. It introduces System1-Arena, a benchmark with 92k questions across 105 tasks and 138 closed-loop environments, and evaluates 30 models. The largest System-1 models (Jev, Decision-2.0-Vega-27B) match or exceed fast closed LLMs (GPT-6 Luna) on non-agentic accuracy, with better-calibrated probabilities at a quarter of the latency. Reasoning LLMs (Claude Opus 5.5) achieve the highest accuracy at about 70x the cost. In closed-loop agentic tasks, large open LLMs succeed most often; small System-1 models rarely succeed.
What carries the argument. System1-Arena's decision interface, which represents every example and environment step as a request containing a state and typed questions (boolean, choice, score). This standardizes input across all model families and enables probability scoring. A real-time protocol advances the environment during model inference to measure the value of fast decisions, and a method for obtaining decision probabilities from open autoregressive and diffusion LLMs through answer-token log-probabilities is provided.
Core claim. The paper establishes that the largest System-1 models can replace generative LLMs for many static decision tasks without sacrificing accuracy, while providing better probability calibration at much lower cost. It demonstrates this by comparing models on a shared decision interface across diverse tasks, finding that Decision-2.0-Vega-27B and Jev achieve 73.0% and 72.6% accuracy respectively, comparable to GPT-6 Luna's 71.2%, with Brier scores of 0.343 and 0.349 versus 0.461. The paper also discovers that step accuracy does not predict closed-loop success, and that forecasting remains a difficult domain for System-1 models, with no model exceeding 61.6% accuracy.
The premise it rests on. The benchmark uses a fixed, single prompt per model family, which may not be equally fair to all models, and 33% of task instructions were written by the authors, inheriting label noise from sources and potentially affecting model rankings.
If the paper is right (Pith's inferences, not the authors')
System-1 models can be deployed for high-volume, low-latency decisions like phishing detection, safety moderation, and routing, reducing cost and latency without sacrificing accuracy.
The benchmark provides a standard for comparing System-1 and System-2 models, enabling informed routing decisions between them based on task type.
The finding that step accuracy does not correlate with closed-loop success suggests that evaluating agents requires full environment interaction, not static step accuracy.
Forecasting tasks remain challenging for System-1 models, indicating that some domains still require the broader knowledge or reasoning of larger LLMs.
The benchmark could be extended to multi-modal decisions (image, audio) to test System-1 models in richer domains, potentially revealing where fast, single-pass processing fails further.
The fixed prompt per family may disadvantage some models; per-task prompt tuning could change rankings, especially for instruction-sensitive tasks where wording strongly influences performance.
The poor calibration of answer-token probabilities in open LLMs (e.g., Gemma-4-31B with Brier score 0.474 despite 73.9% accuracy) suggests that using these probabilities for decision thresholds may be unreliable, a gap that System-1 models fill.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human. System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. Which decisions can System-1 models handle, and which require generative reasoning? We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-16
The authors' main claim, as the classifier picked it out 0.81
“The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.48 · human participants 0.09 · controlled comparison 0.57 · randomized assignment 0.03 · states quantitative result 0.99 · compares with prior results 0.91 · mentions code or data availability 0.08
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human. | background 1.00 | 0.06 | 0.02 | asserted 0.84 |
| System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. | background 1.00 | 0.09 | 0.03 | asserted 1.00 |
| Which decisions can System-1 models handle, and which require generative reasoning? | gap 0.74 | 0.04 | 0.03 | unclear 0.99 |
| We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. | contribution 0.97 | 0.93 | 0.04 | asserted 0.99 |
| It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. | method 0.88 | 0.71 | 0.03 | asserted 1.00 |
| It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. | method 0.85 | 0.71 | 0.03 | asserted 1.00 |
| We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. | method 0.97 | 0.88 | 0.04 | asserted 0.92 |
| The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. | result 1.00 | 0.94 | 0.35 | asserted 1.00 |
| A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. | result 0.99 | 0.87 | 0.24 | asserted 1.00 |
| In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed. | result 1.00 | 0.92 | 0.16 | asserted 1.00 |
What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.97
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-16: 1 request, 7,589 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (5)
Figures are the authors', reproduced from the paper.
References (31)
- [2] ABO: dataset and benchmarks for Real-World 3D object understanding. In2022 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), page 21094–21104. IEEE. Convai Innovations. 2026. Laya. https://laya. convaiinnovations.com/. Nick Craswell, Bhaskar Mitra, Emine... on PithPith's reviewarXiv
- [6] Vulnerability detection with code language models: How far are we? In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), page 1729–1741. IEEE. Tin Dizdarevic, Ravi Hammond, Tobias Gessler, Anisoara Calinescu, Jonathan Cook, Matteo Gallici, Andrei Lupu, ... on Pith
- [7] e-CARE: a new dataset for exploring explain- able causal reasoning. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 432–446, Dublin, Ireland. Association for Computational Lin- guistics. Esin Durmus, Kari... on PithPith's reviewarXiv
- [9] How to evaluate reward models for RLHF. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, on Pith
- [10] OpenReview.net. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.Transactions of the Association for Computational Linguistics, 9:346– 361. Sha... on PithPith's reviewarXiv
- [12] Measuring Massive Multitask Language Un- derstanding.arXiv preprint arXiv:2009.03300. Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021b. CUAD: an Expert-Annotated NLP dataset for legal contract review. InProceedings of the Neural Information Processing Systems Tr... on PithPith's reviewarXiv
- [13] CoSQA: 20,000+ web queries for code search and question answering. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 5690–5700, O... on PithPith's reviewarXiv
- [14] Language Models (Mostly) Know What They Know.arXiv preprint arXiv:2207.05221. Daniel Kahneman. 2011.Thinking, Fast and Slow. Far- rar, Straus and Giroux. Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. 2023. WiCE: Real-World entailment for claims in Wikipedia. ... on PithPith's reviewarXiv
- [16] ATBench: a diverse and realistic agent trajec- tory benchmark for safety evaluation and diagnosis. Preprint, arXiv:2604.02022. Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sun- daresa... on PithPith's reviewarXiv
- [20] WDC products: A Multi-Dimensional entity matching benchmark. InProceedings 27th Interna- tional Conference on Extending Database Technol- ogy, EDBT 2024, Paestum, Italy, March 25 - March 28, pages 22–33. OpenProceedings.org. Denis Peskov, Benny Cheng, Ahmed Elgohary, Joe Bar- ... on Pith
- [21] The text anonymization benchmark (TAB): A dedicated corpus and evaluation framework for text anonymization.Computational Linguistics, 48(4):1053–1101. Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba on Pith
- [22] VirtualHome: simulating household activi- ties via programs. In2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, page 8494–8502. IEEE. Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hanna Haj... on Pith
- [23] Shopping queries dataset: A Large-Scale ESCI benchmark for improving product search.CoRR, abs/2206.06588. Theodoros Rekatsinas, Xu Chu, Ihab F. Ilyas, and Christopher Ré. 2017. HoloClean: holistic data re- pairs with probabilistic inference.Proceedings of the VLDB Endowment, 1... on PithPith's reviewarXiv
- [25] Overview of the TREC 2022 clinical trials track. InProceedings of the Thirty-First Text RE- trieval Conference, TREC 2022, online, November 15-19, 2022, volume 500-338 ofNIST Special Publi- cation. National Institute of Standards and Technol- ogy (NIST). Melissa Roemmele, Cosm... on PithPith's reviewarXiv
- [26] StepGame: a new benchmark for robust Multi-Hop spatial reasoning in texts.Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):11321–11329. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2020. ALFWorld: ... on PithPith's reviewarXiv
- [27] OpenReview.net. Joseph Suh, Erfan Jahanparast, Suhong Moon, Minwoo Kang, and Serina Chang. 2025. Language model Fine-Tuning on scaled survey data for predicting dis- tributions of public opinions. InProceedings of the 63rd Annual Meeting of the Association for Compu- tational ... on PithPith's reviewarXiv
- [28] Gymnasium: A Standard Interface for Rein- forcement Learning Environments.arXiv preprint arXiv:2407.17032. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multi- hop questions via single-hop question composition. Transactions of the A... on PithPith's reviewarXiv
- [30] Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning.arXiv preprint arXiv:1910.10897. Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen on PithPith's reviewarXiv
- [31] MultiWOZ 2.2 : A dialogue dataset with additional annotation corrections and state tracking baselines. InProceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 109–117, Online. Association for Computa- tional Linguistics. Qunhong Zeng, Yuxi... on PithPith's reviewarXiv
- [2011] Learning to grade short answer questions us- ing semantic similarity measures and dependency graph alignments. InProceedings of the 49th An- nual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 752–762, Portland, Oregon, USA. Associ... on Pith
show 11 more
- [2013] SemEval-2013 task 7: The joint student re- sponse analysis and 8th recognizing textual entail- ment challenge. InSecond Joint Conference on Lexi- cal and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Work- shop on Semantic Evaluation (SemEv... on Pith
- [2015] Matching HTML tables to DBpedia. InPro- ceedings of the 5th International Conference on Web Intelligence, Mining and Semantics, WIMS ’15, page 1–6. ACM. Kirk Roberts, Dina Demner-Fushman, Ellen M. V oorhees, Steven Bedrick, and William R. Hersh on Pith
- [2018] Deep learning for entity matching: A design space exploration. InProceedings of the 2018 Inter- national Conference on Management of Data, SIG- MOD/PODS ’18, page 19–34. ACM. Niklas Muennighoff, Qian Liu, Armel Randy Ze- baze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam ... on PithPith's reviewarXiv
- [2019] Latent retrieval for weakly supervised open domain question answering. InProceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 6086–6096, Florence, Italy. Association for Computational Linguistics. Edouard Leurent. 2018. An environmen... on Pith
- [2020] GoEmotions: A dataset of Fine-Grained emo- tions. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040–4054, Online. Association for Computational Linguistics. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wan... on Pith
- [2021] OpenReview.net. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt on Pith
- [2022] CEFR-Based sentence difficulty annotation and assessment. InProceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 6206–6219, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics. Gaurav Arora, Chirag Jain, Manas... on PithPith's reviewarXiv
- [2023] PlanBench: an extensible benchmark for eval- uating large language models on planning and rea- soning about change. InAdvances in Neural Infor- mation Processing Systems 36, NeurIPS 2023, page 38975–38987. Neural Information Processing Sys- tems Foundation, Inc. (NeurIPS). Ber... on PithPith's reviewarXiv
- [2024] Let’s verify step by step. InThe Twelfth In- ternational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open- Review.net. Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. 2025... on PithPith's reviewarXiv
- [2025] TRAIL: trace reasoning and agentic issue localization.Preprint, arXiv:2505.08638. DiffusionGemma Team, Adrien Ali Taïga, James Assiene, Daniele Calandriello, Rahma Chaabouni, João Gante, Tamara von Glehn, Nate Keating, Chris Knutsen, Martin Kukla, Tianlin Liu, Ivan Lobov, et al on PithPith's reviewarXiv
- [2026] DiffusionGemma Technical Report.arXiv preprint arXiv:2608.00146. Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. 2021. Few-NERD: A few-shot named entity recognition dataset. InProceedings of the 59th An- nual Meeting of the... on PithPith's reviewarXiv
Cited by 7 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- Entity Matching using Decision Models cs.CL · Oct 2026
- Joint or Per-Option? Compact Encoders for Closed-Set Decisions from 17 Million to 1 Billion Parameters cs.CL · Oct 2026
- From Large Language Models to Efficient System One Models via Adaptive Computation cs.CL · Nov 2026
- SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models cs.CL · Nov 2026
- Nudging Decision Models: Do Digital Nudges Steer Fast, Generation-Free AI Agents? cs.AI · Nov 2026
- Jev with Foresight: Safe Language-Guided Autonomous Driving via a Language World Model cs.RO · Nov 2026
- Understanding Jev-Style System One Models: Does Structure Reduce Selection Bias? cs.CL · Oct 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment