REVIEW 1 cited by
Can Transformers Reason in Fragments of Natural Language?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
State-of-the-art deep-learning-based approaches to Natural Language Processing (NLP) are credited with various capabilities that involve reasoning with natural language texts. In this paper we carry out a large-scale empirical study investigating the detection of formally valid inferences in controlled fragments of natural language for which the satisfiability problem becomes increasingly complex. We find that, while transformer-based language models perform surprisingly well in these scenarios, a deeper analysis re-veals that they appear to overfit to superficial patterns in the data rather than acquiring the logical principles governing the reasoning in these fragments.
Forward citations
Cited by 1 Pith paper
-
Logical Judgments Under Pressure: Diagnosing Syllogistic Stability with Learned Soft Prefixes
Learned soft prefixes reliably flip correct syllogistic judgments in LLMs, transferring across unseen forms and interfaces and behaving mainly as a broad answer preference rather than a transferable logical operation.
Discussion (0). Sign in to comment.