REVIEW 3 cited by
Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The success of multilingual pre-trained models is underpinned by their ability to learn representations shared by multiple languages even in absence of any explicit supervision. However, it remains unclear how these models learn to generalise across languages. In this work, we conjecture that multilingual pre-trained models can derive language-universal abstractions about grammar. In particular, we investigate whether morphosyntactic information is encoded in the same subset of neurons in different languages. We conduct the first large-scale empirical study over 43 languages and 14 morphosyntactic categories with a state-of-the-art neuron-level probe. Our findings show that the cross-lingual overlap between neurons is significant, but its extent may vary across categories and depends on language proximity and pre-training data size.
Forward citations
Cited by 3 Pith papers
-
Brains and language models converge on a shared conceptual space across different languages
Language models and human brains encode story meaning in a shared conceptual space that generalizes across English, Chinese, and French.
-
Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers
By patching averaged concept representations across languages into transformer models, the authors show the models can still translate and even describe those concepts, supporting the idea of language-agnostic concept...
-
Polysemy of Synthetic Neurons Towards a New Type of Explanatory Categorical Vector Spaces
A GPT2-XL analysis reports that a neuron's highest-activation tokens are also the ones most similar to multiple categorical subclusters, offered as evidence for an intra-neuronal vector-space view of polysemy.
Discussion (0). Continue with ORCID to comment.