REVIEW 6 cited by
Transformers as Soft Reasoners over Language
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Beginning with McCarthy's Advice Taker (1959), AI has pursued the goal of providing a system with explicit, general knowledge and having the system reason over that knowledge. However, expressing the knowledge in a formal (logical or probabilistic) representation has been a major obstacle to this research. This paper investigates a modern approach to this problem where the facts and rules are provided as natural language sentences, thus bypassing a formal representation. We train transformers to reason (or emulate reasoning) over these sentences using synthetically generated data. Our models, that we call RuleTakers, provide the first empirical demonstration that this kind of soft reasoning over language is learnable, can achieve high (99%) accuracy, and generalizes to test data requiring substantially deeper chaining than seen during training (95%+ scores). We also demonstrate that the models transfer well to two hand-authored rulebases, and to rulebases paraphrased into more natural language. These findings are significant as it suggests a new role for transformers, namely as limited "soft theorem provers" operating over explicit theories in language. This in turn suggests new possibilities for explainability, correctability, and counterfactual reasoning in question-answering.
Forward citations
Cited by 6 Pith papers
-
DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoning
DecompSR is a large, symbolically verified benchmark dataset and generation framework that independently varies productivity, substitutivity, overgeneralisation, and systematicity to probe compositional multihop spati...
-
Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL
DRER rewards CoT trajectories that increase the model's likelihood of the correct answer, plus a length penalty, and the new LogicTree benchmark reportedly lifts a 7B model's average accuracy from 0.13 to 0.60.
-
Rule2Text: A Framework for Generating and Evaluating Natural Language Explanations of Knowledge Graph Rules
Rule2Text generates and evaluates natural language explanations of knowledge graph rules, finding that chain-of-thought prompting with entity types works best and that fine-tuning Zephyr on LLM-built ground truth shar...
-
Rule2Text: Natural Language Explanation of Logical Rules in Knowledge Graphs
LLMs generate mostly correct and clear explanations of knowledge-graph logical rules, and combining chain-of-thought prompting with entity type hints improves quality.
-
Mitigating Spurious Correlations in LLMs via Causality-Aware Post-Training
Fine-tuning a 3B LLM on randomly symbolized reasoning questions reduces spurious-correlation failures and improves OOD accuracy on CLadder and PrOntoQA.
-
Leveraging Large Language Models for Bengali Math Word Problem Solving with Chain of Thought Reasoning
A new Bengali math word problem dataset translated from GSM8K is benchmarked with chain-of-thought prompting, yielding 88% accuracy with LLaMA-3.3 70B on a 1,000-sample test subset.
Discussion (0). Sign in to comment.