REVIEW 4 cited by
Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languages
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Human bilinguals often use similar brain regions to process multiple languages, depending on when they learned their second language and their proficiency. In large language models (LLMs), how are multiple languages learned and encoded? In this work, we explore the extent to which LLMs share representations of morphsyntactic concepts such as grammatical number, gender, and tense across languages. We train sparse autoencoders on Llama-3-8B and Aya-23-8B, and demonstrate that abstract grammatical concepts are often encoded in feature directions shared across many languages. We use causal interventions to verify the multilingual nature of these representations; specifically, we show that ablating only multilingual features decreases classifier performance to near-chance across languages. We then use these features to precisely modify model behavior in a machine translation task; this demonstrates both the generality and selectivity of these feature's roles in the network. Our findings suggest that even models trained predominantly on English data can develop robust, cross-lingual abstractions of morphosyntactic concepts.
Forward citations
Cited by 4 Pith papers
-
Semantic Primes as Explanans for Emotion in Large Language Models
NSM semantic primes are more recoverable, more causally effective, and behaviorally more interchangeable with emotions than appraisal dimensions in four instruction-tuned LLMs.
-
Rethinking Cross-lingual Gaps from a Statistical Viewpoint
Cross-lingual accuracy gaps in LLMs are dominated by higher response variance in target languages, not missing knowledge; ensembling and variance-reduction prompts shrink the gap.
-
The Emergence of Abstract Thought in Large Language Models Beyond Any Language
Across 20 open LLMs, shared multilingual neurons grow in number and per-neuron importance over release generations, which the authors interpret as evidence of language-agnostic abstract thought and use to guide neuron...
-
How Syntax Specialization Emerges in Language Models
Syntactic specialization in language models emerges gradually during training, concentrates in particular layers, and appears to stabilize after roughly 16 million tokens.
Discussion (0). Sign in to comment.