Pith. sign in

REVIEW 18 cited by

MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.16049 v2 pith:5U7KA5QG submitted 2023-10-24 cs.CL

Zayne Sprague , Xi Ye , Kaj Bostrom , Swarat Chaudhuri , Greg Durrett This is my paper · ORCID
classification cs.CL
keywords reasoningdatasetchain-of-thoughtlanguagelikellmstechniqueswhile
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

While large language models (LLMs) equipped with techniques like chain-of-thought prompting have demonstrated impressive capabilities, they still fall short in their ability to reason robustly in complex settings. However, evaluating LLM reasoning is challenging because system capabilities continue to grow while benchmark datasets for tasks like logical deduction have remained static. We introduce MuSR, a dataset for evaluating language models on multistep soft reasoning tasks specified in a natural language narrative. This dataset has two crucial features. First, it is created through a novel neurosymbolic synthetic-to-natural generation algorithm, enabling the construction of complex reasoning instances that challenge GPT-4 (e.g., murder mysteries roughly 1000 words in length) and which can be scaled further as more capable LLMs are released. Second, our dataset instances are free text narratives corresponding to real-world domains of reasoning; this makes it simultaneously much more challenging than other synthetically-crafted benchmarks while remaining realistic and tractable for human annotators to solve with high accuracy. We evaluate a range of LLMs and prompting techniques on this dataset and characterize the gaps that remain for techniques like chain-of-thought to perform robust reasoning.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Will Scaling Improve Social Simulation with LLMs?

    cs.CL 2026-07 conditional novelty 7.0 of 10

    Using 85 controlled and 35 public LLMs, the authors show social-simulation accuracy generally improves with compute, but some behavioral and low-resource tasks do not scale.

  2. SCOPE and SCION: A Benchmark and an Auditable Reference Pipeline for Schema Induction and Fusion from Text

    cs.AI 2026-05 conditional novelty 6.0 of 10

    A 24-dataset benchmark for inducing schema graphs from raw text, plus an auditable LLM-based pipeline that reports the highest scores on the benchmark's four schema-similarity metrics.

  3. Throttling Web Agents Using Reasoning Gates

    cs.AI 2025-09 conditional novelty 6.0 of 10

    Rebus-based reasoning gates, puzzles built from random word/domain clue sets, impose token costs on LM web agents that are up to 9.2x the generator's cost.

  4. Are LLM Belief Updates Consistent with Bayes' Theorem?

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Larger pretrained language models show higher Bayesian Coherence Coefficient, meaning their in-context credence updates correlate more strongly with the likelihood-based updates Bayes' theorem would predict.

  5. CCQ: Convolutional Code for Extreme Low-bit Quantization in LLMs

    cs.LG 2025-07 conditional novelty 6.0 of 10

    CCQ compresses LLMs to 2.0-2.75 bits per weight using convolutional codes and bit-shift decoding, shrinking 671B-parameter models to under 200GB.

  6. MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidance

    cs.CL 2025-06 conditional novelty 6.0 of 10

    MDBench is a synthetically generated, knowledge-guided benchmark for multi-document QA on which frontier LLMs achieve only about 60% exact match.

  7. How Benchmark Prediction from Fewer Data Misses the Mark

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Benchmark prediction methods mostly work by interpolation among similar models and fail on better, unfamiliar models, where random sampling with an AIPW-style correction is the only consistent improvement.

  8. Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    The monotonicity of token probabilities during initial decoding predicts chain-of-thought gains, enabling dynamic selection between CoT and direct answers.

  9. RICo: Refined In-Context Contribution for Automatic Instruction-Tuning Data Selection

    cs.CL 2025-05 conditional novelty 6.0 of 10

    RICo scores instruction examples by their in-context perplexity effect on an assessment set, then trains a lightweight selector to pick top-scoring data, achieving better benchmark results from 5% to 15% of the original data.

  10. PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian

    cs.CL 2025-02 conditional novelty 6.0 of 10

    PerCul is a Persian cultural story-comprehension benchmark on which the best LLMs lag human performance by 11.3 to 21.3 percentage points.

  11. JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    JustLogic, a synthetic benchmark with high linguistic and argument complexity, shows most LLMs underperform the average human in pure deductive reasoning.

  12. Domain Adaptation of Foundation LLMs for e-Commerce

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Continued pretraining Llama 3.1 on 1 trillion e-commerce tokens produces e-Llama 8B/70B models that improve in-house e-commerce benchmarks by roughly 25-30% while retaining most general-domain accuracy.

  13. AstroMLab 3: Achieving GPT-4o Level Performance in Astronomy with a Specialized 8B-Parameter Large Language Model

    astro-ph.IM 2024-11 conditional novelty 6.0 of 10

    A specialized 8B-parameter language model trained on astronomy literature and synthetic Q&A matches GPT-4o on the AstroMLab-1 astronomy benchmark at a fraction of the cost.

  14. Decision Protocols in Multi-Agent Large Language Model Conversations

    cs.MA 2026-07 conditional novelty 5.0 of 10

    Consensus decision protocols beat voting/judge on knowledge QA for Llama-3 multi-agent chats, while voting and judge win on logic tasks; independent initial drafts raise accuracy and extra voting-time info barely helps.

  15. Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse

    cs.CL 2026-02 unverdicted novelty 5.0 of 10

    Attention-sink weight is recast as an implicit MoE router per head, motivating a sink-aware head-balancing loss that yields small, consistent benchmark gains across three attention variants but rests on a definitional...

  16. Towards Effective Human-in-the-Loop Assistive AI Agents

    cs.CV 2025-07 conditional novelty 5.0 of 10

    AI-assisted AR guidance improved first-trial task success from 20% unassisted to 70% in a 12-participant, 144-session human study, alongside a new multimodal dataset and evaluation framework.

  17. EfficientLLM: Efficiency in Large Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A large-scale benchmark of LLM efficiency techniques finds that every method trades off one resource for another, with the best choice depending on model scale, task, and hardware.

  18. CABS+: Efficient and Scalable Model Merging via Conflict-Aware Sparsification and Adaptive Weight Allocation

    cs.AI 2026-08 conditional novelty 4.0 of 10

    CABS+ prunes task vectors with a conflict-aware, balanced sparsification step and sets merging coefficients through CMA-ES search, improving efficiency and average merged-model performance.

Pith tools