Pith.

Computation and Language · arXiv:2610.21244 v1 · submitted 15 Oct 2026 · 31 references

System1-Arena: Which Decisions Can Be Left to System 1?

Tianyu Chen, Xinyang Liu, Ruyi An, Zhendong Wang, Mingyuan Zhou

The paper says

“Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human.”

more of the abstract

System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. Which decisions can System-1 models handle, and which require generative reasoning? We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 16 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

The passes disagreed. Reader: holds up. Skeptic: holds up only with conditions. Referee: major revision.

A careful, large-scale benchmark that gives practical answers about where System-1 models are ready—and where they are not. The main finding is robust despite a minor confound in prompting. This line is wrong

3 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 31 references say.

The paper says, at Section 4, Appendix A.1 vs A.2

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

Prompt asymmetry between closed and open LLMs. Closed LLMs receive a JSON probability output prompt (Appendix A.1: 'Reply with a single JSON object and nothing else'), while open LLMs receive a simpler label-selection prompt (Appendix A.2: 'Reply with the label of your answer only').

more of objection 1

The paper states one fixed prompt per family (Section 4), but this asymmetry could systematically affect accuracy, especially given the small reported gaps (1.4–1.8 pp) between System-1 models and closed LLMs such as GPT-6 Luna. The paper provides no control experiment testing closed LLMs with a simpler label-output prompt (even if token probabilities are not available, a label-selection prompt with post-hoc calibration could be compared). Without this control, the claim that System-1 models 'match or exceed' fast closed LLMs is not robustly established.

The paper says, at Section 7 (Limitations)

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.92):

“The models that sample their answers got one draw per request, so run-to-run variation is not measured. 8 Ethical Considerations System1-Arena reuses public datasets and environ- ments under their licenses (Table 4); sources whose terms do not allow redistribution are rebuilt locally by a loader and are not redistributed.”

Pith says: Objection 2 of 3 · major

Single draw per request and lack of stability analysis. The authors note that models that sample their answers got one draw per request and that run-to-run variation is not measured. This is relevant because the accuracy differences that underpin the main claim are small (1.4–1.8 pp).

more of objection 2

The paper does not provide confidence intervals or a bootstrap analysis for the overall accuracy rankings in Table 1, nor does it assess how stable the relative ordering of the top models is under repeated evaluation. This weakness is acknowledged but not addressed.

The paper says, at Section 3.4, Table 1

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

Calibration comparison across different probability elicitation methods. The paper reports that System-1 models have lower Brier scores than fast closed LLMs (e.g., Decision-2.0-Vega-27B 0.343 vs GPT-6 Luna 0.461). However, closed LLMs provide probabilities via verbalized JSON oputput, while open LLMs use answer-token probabilities (Section 4).

more of objection 3

These different elicitation methods are known to affect calibration (as the paper itself cites Tian et al., 2023; Xiong et al., 2023). The calibration advantage attributed to System-1 models may be partly a methodological artefact. The paper should discuss this caveat when interpreting the Brier score comparison.

5 minor objections

The paper says, at Abstract

Pith says: minor objection 1 of 5

The phrase 'in real-time mode, the world evolves while a model decides' appears twice in the abstract; consider removing one instance.

The paper says, at Table1

Pith says: minor objection 2 of 5

The row for Jev is labelled 'Jev 1.13'. Clarify whether the '1.13' reffers to the model version; consider removing it from the row label for consistency with other models.

show 3 more

The paper says, at Section 4

Pith says: minor objection 3 of 5

When describing open LLM probability extraction, the paper says 'GPT-OSS requires reasoning, so we read label probabilities at the start of its final answer.' It may help to explicitly note that gpt-oss-20b and gpt-oss-120b are the two models that use this mode.

The paper says, at Figure1

Pith says: minor objection 4 of 5

The axes labels are very small; consider increasing font size for better readability, especially in the scatter plot of accuracy vs latency.

The paper says, at Appendix B

Pith says: minor objection 5 of 5

The description of 'budget 0' for thinking modes is helpful but could be moved into the main text (Section 5) since it explains the curve origins in Figure 4.

What would settle it

Re-evaluate GPT-6 Luna on a random 5,000-question subset of the non-agentic tasks using a label-only prompt (identical in format to the open LLM prompt, Appendix A.2, returning a single label). Obtain the model's confidence as the softmax probability of the first token of the answer label, available through the GitHub Copilot API if logprobs are supported, or through a separate API (e.g., direct endpoint). Compare accuracy to the JSON-prompt result. If the label-only accuracy exceeds 71.2% by more than 1 percentage point, the headline claim weakens.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 16 Oct 2026

  1. Reader holds up confidence high
  2. Skeptic partly agrees: says it holds up only with conditions 1 min 30 s · 60,991 tokens read, 9,042 written Closed LLM accuracy may be depressed by the JSON probability output format; the paper provides no control comparison with a simpler prompt.
  3. Editor wrote the verdict above 18 s · 61,824 tokens read, 1,549 written
  4. Referee major revision: 3 major objections, 5 minor 24 s · 61,349 tokens read, 2,637 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This is the most thorough comparison of System-1 models and generative LLMs on a shared decision interface I have seen. 92k questions, 30 models, 138 environments—the scale alone makes it useful. The real-time mode that advances the environment during inference is a nice touch. The authors are transparent about their limitations: label noise, contamination, run-to-run variation, and the fact that 33% of instructions are author-written (tier C). They check that agentic harnesses are not too easy via uniform-random baselines. The code and data appear to be public. All of this is good practice.

The core result—that the largest System-1 models match fast closed LLMs (e.g., GPT-6 Luna) on non-agentic accuracy with better calibration at one-quarter the latency—holds up under the data presented. The accuracy gaps are small (1.4–1.8 pp), but the pattern is consistent across domains. The paired comparison (Decisions API vs. GPT-6 Luna) shows that the same underlying model loses 2.6 pp when accessed through the structured interface, which actually suggests the JSON prompt is not the cause of Luna's lower score relative to Jev or Decision-2.0-Vega-27B.

That said, the stress-test concern is real: the paper does not control for prompt format. Closed LLMs get a JSON instruction; open and System-1 models get different prompts. The paper acknowledges a fixed prompt per family, but a simple ablation (e.g., asking closed LLMs to output a label instead of JSON, then post-hoc calibrating) would have addressed it directly. Given the small accuracy margins, a prompt change could shift rankings. This is a minor weakness, not a fatal one—the authors are forthright about the limitation, and the paired comparison partially mitigates the worry.

The benchmark is squarely aimed at practitioners who need to decide whether to route a decision to a fast, calibrated model or pay for generative reasoning. Researchers studying System-1 architectures or calibration will also find value. I would cite this as a reference dataset for decision-focused evaluation.

Verdict: deserves a serious referee. I would accept it with a minor comment asking for a control experiment on prompt format, but the paper is already strong enough without it.

The strongest objection

The central claim that System-1 models match or exceed fast closed LLMs (GPT-6 Luna) on non-agentic accuracy rests on an accuracy difference of 1.4–1.8 percentage points (Table 1). Closed LLMs are evaluated through a fixed prompt that asks for probabilities in JSON (Appendix A.1), which is more complex than the label-selection prompt given to open LLMs (Appendix A.2). The paper acknowledges a fixed prompt per family (Section 4) but does not test whether closed LLM accuracy changes when evaluated via a simpler label-output prompt (even if token probabilities are not available, a label-selection prompt with a post-hoc calibration step could be compared). Because the accuracy gap is small, a shift of a few points from a prompt change could eliminate it. The paper's own paired comparison (Figure 5) shows that GPT-6 Luna (71.2%) outperforms the Decisions API (68.6%), which uses the same underlying model through a structured interface; this suggests the JSON format is not obviously harmful, but a direct controlled experiment with alternative prompts is missing. Without this control, the claim of equivalence or superiority is not robustly established.

The reader's weakest_assumption identifies prompt fairness as a limitation, noting the paper's fixed single prompt and the acknowledgement that tier-C instructions were written by the authors. My concern is more specific: it targets the JSON probability output format for closed LLMs as a potential source of bias that could affect the central accuracy comparison. I agree with the reader that this is the weakest area, but I drill down on a concrete mechanism (the JSON output format vs. label-only output) and propose a specific test. The reader's verdict of ACCEPT is reasonable, but I think this concern warrants a CONDITIONAL verdict: the paper should demonstrate that the accuracy ranking is robust to at least one alternative prompt format for closed LLMs. If the authors provide this validation or show that the JSON format does not depress accuracy, the concern is resolved and ACCEPT stands. If they cannot, the central claim of System-1 models matching or exceeding fast closed LLMs is not firmly supported.

The referee's report

If the main claims hold, this benchmark provides a practical tool for deciding when to use fast, non-generative decision models versus slower generative LLMs. The breadth of tasks (105 tasks, 16 domains) and the inclusion of agentic environments with a real-time mode are methodological contributions. The finding that System-1 models match fast closed LLMs on accuracy while offering better calibration and lower latency is practically relevant for deployment in high-volume, latency-sensitive applications. However, the benchmark's utility depends on whether the model comparisons are robust to evaluation design choices, which are not fully controlled.

Pith's reading of the paper (a machine's summary, not the authors')

The paper aims to answer which discrete decisions can be handled by System-1 models—models that return probabilities in a single forward pass—versus generative LLMs. It introduces System1-Arena, a benchmark with 92k questions across 105 tasks and 138 closed-loop environments, and evaluates 30 models. The largest System-1 models (Jev, Decision-2.0-Vega-27B) match or exceed fast closed LLMs (GPT-6 Luna) on non-agentic accuracy, with better-calibrated probabilities at a quarter of the latency. Reasoning LLMs (Claude Opus 5.5) achieve the highest accuracy at about 70x the cost. In closed-loop agentic tasks, large open LLMs succeed most often; small System-1 models rarely succeed.

What carries the argument. System1-Arena's decision interface, which represents every example and environment step as a request containing a state and typed questions (boolean, choice, score). This standardizes input across all model families and enables probability scoring. A real-time protocol advances the environment during model inference to measure the value of fast decisions, and a method for obtaining decision probabilities from open autoregressive and diffusion LLMs through answer-token log-probabilities is provided.

Core claim. The paper establishes that the largest System-1 models can replace generative LLMs for many static decision tasks without sacrificing accuracy, while providing better probability calibration at much lower cost. It demonstrates this by comparing models on a shared decision interface across diverse tasks, finding that Decision-2.0-Vega-27B and Jev achieve 73.0% and 72.6% accuracy respectively, comparable to GPT-6 Luna's 71.2%, with Brier scores of 0.343 and 0.349 versus 0.461. The paper also discovers that step accuracy does not predict closed-loop success, and that forecasting remains a difficult domain for System-1 models, with no model exceeding 61.6% accuracy.

The premise it rests on. The benchmark uses a fixed, single prompt per model family, which may not be equally fair to all models, and 33% of task instructions were written by the authors, inheriting label noise from sources and potentially affecting model rankings.

If the paper is right (Pith's inferences, not the authors')

System-1 models can be deployed for high-volume, low-latency decisions like phishing detection, safety moderation, and routing, reducing cost and latency without sacrificing accuracy.

The benchmark provides a standard for comparing System-1 and System-2 models, enabling informed routing decisions between them based on task type.

The finding that step accuracy does not correlate with closed-loop success suggests that evaluating agents requires full environment interaction, not static step accuracy.

Forecasting tasks remain challenging for System-1 models, indicating that some domains still require the broader knowledge or reasoning of larger LLMs.

The benchmark could be extended to multi-modal decisions (image, audio) to test System-1 models in richer domains, potentially revealing where fast, single-pass processing fails further.

The fixed prompt per family may disadvantage some models; per-task prompt tuning could change rankings, especially for instruction-sensitive tasks where wording strongly influences performance.

The poor calibration of answer-token probabilities in open LLMs (e.g., Gemma-4-31B with Brier score 0.474 despite 73.9% accuracy) suggests that using these probabilities for decision thresholds may be unreliable, a gap that System-1 models fill.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human. System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. Which decisions can System-1 models handle, and which require generative reasoning? We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-16

The authors' main claim, as the classifier picked it out 0.81

“The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.48 · human participants 0.09 · controlled comparison 0.57 · randomized assignment 0.03 · states quantitative result 0.99 · compares with prior results 0.91 · mentions code or data availability 0.08

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human. background 1.00 0.06 0.02 asserted 0.84
System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. background 1.00 0.09 0.03 asserted 1.00
Which decisions can System-1 models handle, and which require generative reasoning? gap 0.74 0.04 0.03 unclear 0.99
We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. contribution 0.97 0.93 0.04 asserted 0.99
It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. method 0.88 0.71 0.03 asserted 1.00
It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. method 0.85 0.71 0.03 asserted 1.00
We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. method 0.97 0.88 0.04 asserted 0.92
The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. result 1.00 0.94 0.35 asserted 1.00
A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. result 0.99 0.87 0.24 asserted 1.00
In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed. result 1.00 0.92 0.16 asserted 1.00

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.97

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-16: 1 request, 7,589 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (5)
Figure 1 of the paper
Figure 1. System-1 models combine fast, accurate decisions with well-calibrated probabilities. The strongest System-1 models match fast closed LLMs in accuracy, with better probability estimates and lower latency. Reasoning LLMs achieve the highest accuracy. See Section 4 for evaluation settings. et al., 2025c). Routing and cascading reduce these costs, but they still rely on generative models (Chen et al., 2023a; Hu et al., 2024). Latency also limits LLM use in robotics, driving, and games, where the world continues to change during inference. System 1 supports fast, automatic judgments, w
Figure 2 of the paper
Figure 2. A decision request from the CVSS task and the answer of Jev. We show two of its nine questions and shorten the instructions and the option descriptions. An LLM gets the same request as text (Appendix A). fighting game. We use a harness that presents each environment’s decisions as typed questions and add a real-time mode that incorporates each model’s measured latency into the environment dynamics. 3 System1-Arena 3.1 Decision requests We represent every static example and environment step as a decision request r = (s, Q) (
Figure 3 of the paper
Figure 3. Task-domain gaps and agentic perfor￾mance. Left: gaps by task domain in percentage points. Right: agentic task performance. Evaluation settings are given in Section 4. 5 Results
Figure 4 of the paper
Figure 4. Thinking can improve accuracy while wors￾ening Brier scores. Accuracy and Brier score on non￾agentic tasks versus the mean number of reasoning to￾kens per question. The zero-token point disables think￾ing, as in
Figure 5 of the paper
Figure 5. Paired comparisons show mixed gains in non-agentic accuracy but consistently lower agentic step accuracy (left); higher cost does not always yield higher accuracy (right). †Base: Qwen3-0.6B-Base. ‡Same LLM, different interface. per request, while local System-1 models take 41– 228 ms. The two GPT-OSS models use reason￾ing and take 183 and 284 ms. Jev takes 321 ms through the gateway and Microsoft-Decision-1 takes 304 ms, while closed LLMs take 1.3–6.7 s. Cost and output tokens. Higher cost does not always yield higher accuracy among hosted models (

Figures are the authors', reproduced from the paper.

References (31)
  1. [2] ABO: dataset and benchmarks for Real-World 3D object understanding. In2022 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), page 21094–21104. IEEE. Convai Innovations. 2026. Laya. https://laya. convaiinnovations.com/. Nick Craswell, Bhaskar Mitra, Emine... on PithPith's reviewarXiv
  2. [6] Vulnerability detection with code language models: How far are we? In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), page 1729–1741. IEEE. Tin Dizdarevic, Ravi Hammond, Tobias Gessler, Anisoara Calinescu, Jonathan Cook, Matteo Gallici, Andrei Lupu, ... on Pith
  3. [7] e-CARE: a new dataset for exploring explain- able causal reasoning. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 432–446, Dublin, Ireland. Association for Computational Lin- guistics. Esin Durmus, Kari... on PithPith's reviewarXiv
  4. [9] How to evaluate reward models for RLHF. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, on Pith
  5. [10] OpenReview.net. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.Transactions of the Association for Computational Linguistics, 9:346– 361. Sha... on PithPith's reviewarXiv
  6. [12] Measuring Massive Multitask Language Un- derstanding.arXiv preprint arXiv:2009.03300. Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021b. CUAD: an Expert-Annotated NLP dataset for legal contract review. InProceedings of the Neural Information Processing Systems Tr... on PithPith's reviewarXiv
  7. [13] CoSQA: 20,000+ web queries for code search and question answering. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 5690–5700, O... on PithPith's reviewarXiv
  8. [14] Language Models (Mostly) Know What They Know.arXiv preprint arXiv:2207.05221. Daniel Kahneman. 2011.Thinking, Fast and Slow. Far- rar, Straus and Giroux. Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. 2023. WiCE: Real-World entailment for claims in Wikipedia. ... on PithPith's reviewarXiv
  9. [16] ATBench: a diverse and realistic agent trajec- tory benchmark for safety evaluation and diagnosis. Preprint, arXiv:2604.02022. Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sun- daresa... on PithPith's reviewarXiv
  10. [20] WDC products: A Multi-Dimensional entity matching benchmark. InProceedings 27th Interna- tional Conference on Extending Database Technol- ogy, EDBT 2024, Paestum, Italy, March 25 - March 28, pages 22–33. OpenProceedings.org. Denis Peskov, Benny Cheng, Ahmed Elgohary, Joe Bar- ... on Pith
  11. [21] The text anonymization benchmark (TAB): A dedicated corpus and evaluation framework for text anonymization.Computational Linguistics, 48(4):1053–1101. Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba on Pith
  12. [22] VirtualHome: simulating household activi- ties via programs. In2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, page 8494–8502. IEEE. Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hanna Haj... on Pith
  13. [23] Shopping queries dataset: A Large-Scale ESCI benchmark for improving product search.CoRR, abs/2206.06588. Theodoros Rekatsinas, Xu Chu, Ihab F. Ilyas, and Christopher Ré. 2017. HoloClean: holistic data re- pairs with probabilistic inference.Proceedings of the VLDB Endowment, 1... on PithPith's reviewarXiv
  14. [25] Overview of the TREC 2022 clinical trials track. InProceedings of the Thirty-First Text RE- trieval Conference, TREC 2022, online, November 15-19, 2022, volume 500-338 ofNIST Special Publi- cation. National Institute of Standards and Technol- ogy (NIST). Melissa Roemmele, Cosm... on PithPith's reviewarXiv
  15. [26] StepGame: a new benchmark for robust Multi-Hop spatial reasoning in texts.Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):11321–11329. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2020. ALFWorld: ... on PithPith's reviewarXiv
  16. [27] OpenReview.net. Joseph Suh, Erfan Jahanparast, Suhong Moon, Minwoo Kang, and Serina Chang. 2025. Language model Fine-Tuning on scaled survey data for predicting dis- tributions of public opinions. InProceedings of the 63rd Annual Meeting of the Association for Compu- tational ... on PithPith's reviewarXiv
  17. [28] Gymnasium: A Standard Interface for Rein- forcement Learning Environments.arXiv preprint arXiv:2407.17032. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multi- hop questions via single-hop question composition. Transactions of the A... on PithPith's reviewarXiv
  18. [30] Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning.arXiv preprint arXiv:1910.10897. Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen on PithPith's reviewarXiv
  19. [31] MultiWOZ 2.2 : A dialogue dataset with additional annotation corrections and state tracking baselines. InProceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 109–117, Online. Association for Computa- tional Linguistics. Qunhong Zeng, Yuxi... on PithPith's reviewarXiv
  20. [2011] Learning to grade short answer questions us- ing semantic similarity measures and dependency graph alignments. InProceedings of the 49th An- nual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 752–762, Portland, Oregon, USA. Associ... on Pith
show 11 more
  1. [2013] SemEval-2013 task 7: The joint student re- sponse analysis and 8th recognizing textual entail- ment challenge. InSecond Joint Conference on Lexi- cal and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Work- shop on Semantic Evaluation (SemEv... on Pith
  2. [2015] Matching HTML tables to DBpedia. InPro- ceedings of the 5th International Conference on Web Intelligence, Mining and Semantics, WIMS ’15, page 1–6. ACM. Kirk Roberts, Dina Demner-Fushman, Ellen M. V oorhees, Steven Bedrick, and William R. Hersh on Pith
  3. [2018] Deep learning for entity matching: A design space exploration. InProceedings of the 2018 Inter- national Conference on Management of Data, SIG- MOD/PODS ’18, page 19–34. ACM. Niklas Muennighoff, Qian Liu, Armel Randy Ze- baze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam ... on PithPith's reviewarXiv
  4. [2019] Latent retrieval for weakly supervised open domain question answering. InProceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 6086–6096, Florence, Italy. Association for Computational Linguistics. Edouard Leurent. 2018. An environmen... on Pith
  5. [2020] GoEmotions: A dataset of Fine-Grained emo- tions. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040–4054, Online. Association for Computational Linguistics. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wan... on Pith
  6. [2021] OpenReview.net. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt on Pith
  7. [2022] CEFR-Based sentence difficulty annotation and assessment. InProceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 6206–6219, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics. Gaurav Arora, Chirag Jain, Manas... on PithPith's reviewarXiv
  8. [2023] PlanBench: an extensible benchmark for eval- uating large language models on planning and rea- soning about change. InAdvances in Neural Infor- mation Processing Systems 36, NeurIPS 2023, page 38975–38987. Neural Information Processing Sys- tems Foundation, Inc. (NeurIPS). Ber... on PithPith's reviewarXiv
  9. [2024] Let’s verify step by step. InThe Twelfth In- ternational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open- Review.net. Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. 2025... on PithPith's reviewarXiv
  10. [2025] TRAIL: trace reasoning and agentic issue localization.Preprint, arXiv:2505.08638. DiffusionGemma Team, Adrien Ali Taïga, James Assiene, Daniele Calandriello, Rahma Chaabouni, João Gante, Tamara von Glehn, Nate Keating, Chris Knutsen, Martin Kukla, Tianlin Liu, Ivan Lobov, et al on PithPith's reviewarXiv
  11. [2026] DiffusionGemma Technical Report.arXiv preprint arXiv:2608.00146. Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. 2021. Few-NERD: A few-shot named entity recognition dataset. InProceedings of the 59th An- nual Meeting of the... on PithPith's reviewarXiv
Cited by 7 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment