REVIEW 7 cited by
ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Recent powerful pre-trained language models have achieved remarkable performance on most of the popular datasets for reading comprehension. It is time to introduce more challenging datasets to push the development of this field towards more comprehensive reasoning of text. In this paper, we introduce a new Reading Comprehension dataset requiring logical reasoning (ReClor) extracted from standardized graduate admission examinations. As earlier studies suggest, human-annotated datasets usually contain biases, which are often exploited by models to achieve high accuracy without truly understanding the text. In order to comprehensively evaluate the logical reasoning ability of models on ReClor, we propose to identify biased data points and separate them into EASY set while the rest as HARD set. Empirical results show that state-of-the-art models have an outstanding ability to capture biases contained in the dataset with high accuracy on EASY set. However, they struggle on HARD set with poor performance near that of random guess, indicating more research is needed to essentially enhance the logical reasoning ability of current models.
Forward citations
Cited by 7 Pith papers
-
KRISTEVA: Close Reading as a Novel Task for Benchmarking Interpretive Reasoning
KRISTEVA is the first close reading benchmark for large language models, and current models still underperform experienced human readers on 10 of its 11 tasks.
-
Length Penalties Make Chain-of-Thought Less Monitorable
Length-penalized RL shortens chain-of-thought while preserving accuracy and hint influence, but preferentially removes the cues that let a monitor detect that influence.
-
From Atomic Evidence to Logical Composition: Structured Compositional Reasoning over Compound Answer Options
A decomposition-plus-ILP framework with contrastive atomic scoring and relative calibration substantially improves LLM accuracy on compound answer options, with the largest gains on NEITHER/NOR.
-
JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models
JustLogic, a synthetic benchmark with high linguistic and argument complexity, shows most LLMs underperform the average human in pure deductive reasoning.
-
BRoverbs -- Measuring how much LLMs understand Portuguese proverbs
BRoverbs lets researchers test whether language models understand Portuguese proverbs; commercial models nearly master it, small models often guess randomly.
-
Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in Large Language Models
Dynamic chunking plus question-aware chunk selection improves long-context QA, but the headline numbers are partly inflated by choosing hyperparameters on the test benchmarks.
-
Thinking About Thinking: SAGE-nano's Inverse Reasoning for Self-Aware Language Models
A 4B-parameter model is claimed to explain its own reasoning through inverse attention analysis, but the paper offers no consistent evidence or artifacts.
Discussion (0). Continue with ORCID to comment.