REVIEW 5 cited by
A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
This study introduces a hypothesis-testing framework to assess whether large language models (LLMs) possess genuine reasoning abilities or primarily depend on token bias. We go beyond evaluating LLMs on accuracy; rather, we aim to investigate their token bias in solving logical reasoning tasks. Specifically, we develop carefully controlled synthetic datasets, featuring conjunction fallacy and syllogistic problems. Our framework outlines a list of hypotheses where token biases are readily identifiable, with all null hypotheses assuming genuine reasoning capabilities of LLMs. The findings in this study suggest, with statistical guarantee, that most LLMs still struggle with logical reasoning. While they may perform well on classic problems, their success largely depends on recognizing superficial patterns with strong token bias, thereby raising concerns about their actual reasoning and generalization abilities. Codes and data are open-sourced at https://github.com/bowen-upenn/llm_token_bias.
Forward citations
Cited by 5 Pith papers
-
Implicit Reasoning Steering via Concept Chaining
Reinforcement-learning-optimized concept-chain paragraphs covertly steer language-model multiple-choice preferences after continued pretraining, with far lower detectability than direct paraphrases.
-
Understanding the Ability of LLMs to Handle Character-Level Perturbation
LLMs remain surprisingly accurate on math and coding when invisible Unicode noise is inserted after every character, with robustness driven by implicit internal denoising and, for some models, explicit rewriting in ch...
-
Beyond Statistical Learning: Exact Learning Is Essential for General Intelligence
Reliable deductive reasoning in AI requires replacing average-case statistical objectives with the exact learning criterion of universal correctness, a thesis supported by sample-complexity lower bounds showing statis...
-
Do Large Language Models Reason Causally Like Us? Even Better?
GPT-4o, Gemini-Pro, and Claude show less associative bias than human reasoners on collider graphs, making them more normatively aligned in likelihood judgments, but none fully demonstrates explaining away.
-
Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspective
A symbol-remapping benchmark shows that LLMs' arithmetic and symbolic reasoning accuracy drops sharply when familiar digits and operators are replaced, revealing heavy reliance on memorized tokens rather than abstract rules.
Discussion (0). Continue with ORCID to comment.