REVIEW 46 cited by
LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Machine reading is a fundamental task for testing the capability of natural language understanding, which is closely related to human cognition in many aspects. With the rising of deep learning techniques, algorithmic models rival human performances on simple QA, and thus increasingly challenging machine reading datasets have been proposed. Though various challenges such as evidence integration and commonsense knowledge have been integrated, one of the fundamental capabilities in human reading, namely logical reasoning, is not fully investigated. We build a comprehensive dataset, named LogiQA, which is sourced from expert-written questions for testing human Logical reasoning. It consists of 8,678 QA instances, covering multiple types of deductive reasoning. Results show that state-of-the-art neural models perform by far worse than human ceiling. Our dataset can also serve as a benchmark for reinvestigating logical AI under the deep learning NLP setting. The dataset is freely available at https://github.com/lgw863/LogiQA-dataset
Forward citations
Cited by 46 Pith papers
-
Efficient Decentralized Multi-task Dataset Valuation via Model Merging
Task-arithmetic model merging approximates multi-task coalition utilities well enough to recover Dataset Shapley rankings privately and without retraining.
-
R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation
A 3,068-prompt benchmark with per-instance Q&A scoring shows that current text-to-image models, including reasoning-enhanced ones, handle reasoning-driven prompts poorly, with mathematical reasoning near zero.
-
Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards
Non-vacuous PAC-Bayes generalization bounds for billion-parameter RLVR models, obtained by a Gumbel-max reparameterization and aggressive TinyLoRA distillation/quantization, are claimed for four tasks.
-
PACE: A Proxy for Agentic Capability Evaluation
PACE builds proxy benchmarks from non-agentic instances via relevance and global selection plus regression to predict agentic scores with MAE under 4%, Spearman correlation above 0.80, and 85% ranking accuracy at unde...
-
What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs
VLM accuracy can be predicted from a scalar capability score derived from LLM text benchmarks plus multimodal data volume via a fitted transfer-absorption scaling law.
-
Reasoning or Fluency? Dissecting Probabilistic Confidence in Best-of-N Selection
Selection by log-likelihood, entropy, or self-certainty chooses correct reasoning traces almost as well after inter-step dependencies are removed, implying the scores track fluency and priors, not reasoning.
-
DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone
A masked diffusion language model built on a bidirectional Mamba backbone matches Transformer-based denoisers on quality while decoding with near-linear time scaling.
-
BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining
A bilevel optimization method ranks pretraining data by training a small proxy model on weighted samples, yielding modest downstream-task gains without external pretrained models.
-
Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL
DRER rewards CoT trajectories that increase the model's likelihood of the correct answer, plus a length penalty, and the new LogicTree benchmark reportedly lifts a 7B model's average accuracy from 0.13 to 0.60.
-
Language Models Improve When Pretraining Data Matches Target Tasks
Ranking pretraining documents by similarity to benchmark training examples (BETR) yields consistent benchmark gains and a 2.1x compute multiplier over DCLM-Baseline.
-
GeLaCo: An Evolutionary Approach to Layer Compression
Evolutionary search over layer-merging configurations, scored by module-wise activation similarity, yields competitive LLM compression and the first size-quality Pareto fronts.
-
Does Learning Mathematical Problem-Solving Generalize to Broader Reasoning?
Long chain-of-thought and RL training on math problems improves general reasoning benchmarks, while short chain-of-thought math fine-tuning often degrades performance.
-
Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning
Chameleon uses kernel ridge leverage scores on domain embeddings to set LLM training-mixture weights, matching DoGE-level pretraining quality at roughly one fifth the compute and improving finetuning perplexity.
-
Advantageous Parameter Expansion Training Makes Better Large Language Models
APEX expands high-activation parameters into low-activation ones during LLM training and reports gains, but the continued pre-training evidence is confounded by different learning rate schedules.
-
VisualSphinx: Large-Scale Synthetic Vision Logic Puzzles for RL
LLM-derived rules, genetic expansion, and program-drawn images produce 660K visual logic puzzles; GRPO training on them lifts a VLM's accuracy on those puzzles and modestly on MathVista.
-
UAQFact: Evaluating Factual Knowledge Utilization of LLMs on Unanswerable Questions
A new bilingual benchmark ties unanswerable questions to Wikidata facts and shows that LLMs often store the relevant knowledge yet fail to use it to recognize unanswerability.
-
GRAPE: Optimize Data Mixture for Group Robust Multi-target Adaptive Pretraining
GRAPE uses a minimax group-DRO scheme to reweight both source domains and target tasks during pretraining, improving multi-task reasoning and low-resource language modeling.
-
RICo: Refined In-Context Contribution for Automatic Instruction-Tuning Data Selection
RICo scores instruction examples by their in-context perplexity effect on an assessment set, then trains a lightweight selector to pick top-scoring data, achieving better benchmark results from 5% to 15% of the original data.
-
Computational Reasoning of Large Language Models
TMBench measures LLM computational reasoning by having models simulate m-tag systems step by step, and its pass rates correlate with AIME2024, MATH500, GPQA Diamond, and MMLU Pro scores across 12 leading models.
-
MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections
Dynamic multiway dense connections across Transformer layers let a 2.8B model match a 6.9B model's pretraining quality and few-shot performance.
-
DarwinLM: Evolutionary Structured Pruning of Large Language Models
DarwinLM uses evolutionary search with training-aware offspring selection to prune LLMs, beating ShearedLlama with 5x less post-training data.
-
Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining
A fully online, loss-based reweighting scheme that down-weights low-loss samples during LLM pretraining yields small average benchmark gains at 1.4B and 7B scale, together with a convergence bound under convexity and ...
-
Fine, I'll Merge It Myself: A Multi-Fidelity Framework for Automated Model Merging
An automated multi-fidelity search framework discovers layer-wise and depth-wise model merging recipes that improve single- and multi-objective LLM reasoning performance without retraining.
-
ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning
On 1,000 generated logic grid puzzles, LLM accuracy drops sharply as search-space size and Z3 conflict count grow, and neither larger models, more samples, nor longer reasoning chains fully overcome the drop.
-
MM-IQ: Benchmarking Human-Like Abstraction and Reasoning in Multimodal Models
The MM-IQ benchmark shows state-of-the-art multimodal models score 33% on abstract visual reasoning puzzles versus 25% chance and 51% for humans.
-
Lightweight and Post-Training Structured Pruning for On-Device Large Lanaguage Models
COMP prunes LLMs post-training by removing redundant layers and then neurons ranked by a matrix condition-number metric, recovering accuracy via mask tuning.
-
JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models
JustLogic, a synthetic benchmark with high linguistic and argument complexity, shows most LLMs underperform the average human in pure deductive reasoning.
-
NExtLong: Toward Effective Long-Context Training without Long Documents
Interleaving hard negative distractors between chunks of short documents improves long-context language model performance on HELMET and RULER.
-
Mixture of Hidden-Dimensions Transformer
MoHD selectively activates shared and token-specific sub-dimensions of the hidden state, matching or beating dense Transformers with fewer active parameters.
-
Cross-Modal Consistency in Multimodal Large Language Models
GPT-4V answers identical questions much less accurately when they are presented as images than as text, even when it can extract the image content nearly perfectly.
-
Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory
A pretrained parametric memory module, taught to copy nearest-neighbor retrieval for next-token prediction, lets small frozen LMs match or beat much larger LMs at the same total parameter budget.
-
DeepResearch-9K: A Challenging Benchmark Dataset of Deep-Research Agent
A synthetic benchmark of 9,000 multi-hop web-research questions with difficulty tiers and teacher-generated search trajectories, plus an open-source RL training framework that reportedly lets 3B-parameter agents beat ...
-
Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency
Setting the feature dimension of each linear-attention layer proportional to the estimated degrees of freedom of its input kernel improves distilled model accuracy without increasing total inference cost.
-
GTA: Grouped-head latenT Attention
GTA reuses attention scores across grouped heads and stores compressed latent values, roughly matching GQA-level quality in sub-1B models while reducing cache and compute.
-
DONOD: Efficient and Generalizable Instruction Fine-Tuning for LLMs via Model-Intrinsic Dataset Pruning
A model-intrinsic data pruning method using two weight-change metrics and TOPSIS reportedly improves instruction fine-tuning accuracy and robustness while using 70-80% less data.
-
Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models
An attention variant that compresses Key heads harder than Value heads and widens Query heads yields up to 33.36% faster long-context attention, powering a system-domain LLM that reportedly outperforms GPT-4 on the ne...
-
Lillama: Large Language Models Compression via Low-Rank Feature Distillation
Lillama compresses LLMs by SVD-initialized low-rank layers trained with a local Teacher plus Student activation distillation loss, achieving 20-40% parameter reduction with only 13 million calibration tokens.
-
GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI
GMAI-VL-5.5M is a new 5.5M-sample medical image-text dataset built from 219 datasets via GPT-4o annotation-guided generation, and GMAI-VL is a three-stage LLaVA-style model reporting SOTA numbers, though the evaluatio...
-
Understanding Chain-of-Thought in LLMs through Information Theory
A chain-of-thought step's information gain, estimated by a fine-tuned supervisor model, can identify the step where an LLM's reasoning first diverges from the correct answer.
-
Sparse Upcycling: Inference Inefficient Finetuning
Sparse upcycling beats continued pretraining on quality by up to roughly 20 percent at matched compute, but cut serving throughput by 34 to 44 percent in vLLM benchmarks.
-
LLM Data Selection and Utilization via Dynamic Bi-level Optimization
A bi-level-learned per-batch reweighting model improves average downstream accuracy of LLMs pre-trained on selected data by about one to two points, but without error bars or code the effect size is uncertain.
-
ReasonBridge: Efficient Reasoning Transfer from Closed to Open-Source Language Models
ReasonBridge distills 1,000 curated reasoning traces from a closed model into open-source models via hierarchical adapters and guided inference, claiming up to 23% benchmark gains.
-
Reason from Future: Reverse Thought Chain Enhances LLM Reasoning
A prompting method that alternates backward and forward reasoning improves small LLM accuracy on math and search tasks and reduces the number of visited search states.
-
Can OpenAI o1 outperform humans in higher-order cognitive thinking?
The study claims o1-preview surpasses human averages on several higher-order thinking tests, but post-hoc benchmark choices, missing statistics, and unaddressed training-data contamination make the claim unreliable.
-
Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey
A comprehensive review that categorizes methods for shortening and adaptively triggering chain-of-thought reasoning in large language models.
-
Thinking About Thinking: SAGE-nano's Inverse Reasoning for Self-Aware Language Models
A 4B-parameter model is claimed to explain its own reasoning through inverse attention analysis, but the paper offers no consistent evidence or artifacts.
Discussion (0). Continue with ORCID to comment.