By patching averaged concept representations across languages into transformer models, the authors show the models can still translate and even describe those concepts, supporting the idea of language-agnostic concept representations.
What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological properties
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Multilingual sentence encoders have seen much success in cross-lingual model transfer for downstream NLP tasks. Yet, we know relatively little about the properties of individual languages or the general patterns of linguistic variation that they encode. We propose methods for probing sentence representations from state-of-the-art multilingual encoders (LASER, M-BERT, XLM and XLM-R) with respect to a range of typological properties pertaining to lexical, morphological and syntactic structure. In addition, we investigate how this information is distributed across all layers of the models. Our results show interesting differences in encoding linguistic variation associated with different pretraining strategies.
fields
cs.CL 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers
By patching averaged concept representations across languages into transformer models, the authors show the models can still translate and even describe those concepts, supporting the idea of language-agnostic concept representations.