Memorized facts in fine-tuned LLMs often sit off the mid-layer reasoning path; relocating those representations recovers most multi-hop generalization failures.
Do llamas work in english? on the latent language of multilingual transformers
21 Pith papers cite this work, alongside 36 external citations. Polarity classification is still indexing.
21
Pith papers citing it
36
external citations · OpenAlex
representative citing papers
Low-resource safety failures are action failures because the harmfulness representation transfers but the decision calibration does not; this is fixed by recalibrating a high-resource gate with 1-4 target-language examples.
Auto-interpretation labels for SAE features generalize poorly across languages and scripts, missing the same semantic content up to 4x more often in Serbian than English and more in Cyrillic than Latin despite deterministic transliteration.
An 8B autoregressive LM implements a language-switching backdoor via a three-phase circuit with early trigger composition, orthogonal mid-layer propagation, and final-layer MLP conversion, routed through a single-position serial bottleneck.
Parallel-SFT mixes parallel programs across languages during SFT to produce more transferable RL initializations, yielding better zero-shot generalization to unseen programming languages.
Cross-lingual prompt exploration improves factual recall and consistency in LLMs across 17 languages more efficiently than native-language scaling.
Activation steering on early layers improves diversity of synthetic data for low-resource languages and often boosts downstream classifier performance compared to non-steered prompting.
LLMs exhibit higher perplexity on far-right and nationalist party texts than social-democratic ones, consistent across models and languages with correlation to translation metrics.
Bucket-Level MOO reformulates multilingual fine-tuning as localized multi-objective optimization and proves it enforces a tighter Pareto stationarity condition while improving cross-lingual performance on four LLMs.
Unlearning in multilingual LLMs suppresses rather than erases knowledge in later layers, with transfer varying by language similarity and reversible via inference-time steering.
Low-resource languages are structurally more different from English in LLMs than high- or mid-resource ones, and language-specific post-training alters structures while preserving inter-language relationships.
A Bayesian framework decomposes mLLM variance, showing language features explain 79-92% of language identity variance and that model identity vs. benchmark-model interactions dominate differently for understanding versus reasoning tasks.
Four axioms (Causality, Minimality, Separability, Stability) are formalized for latent thought representations; audits of open LLMs on 23 tasks show none satisfy all four and representations add little beyond input embeddings.
Using an asymmetric rank-based measure, the paper maps which layers of large language and vision models carry shared semantic information, finding central-layer peaks for translations and cross-modal pairs, and systematic predictability asymmetries favoring English and larger models.
Parameter alignment strategies substantially reduce forgetting in family-based continual pretraining of multilingual LLMs across 32 languages with minimal impact on language acquisition.
A multilingual self-consistency plus self-critique method raises cultural alignment scores on English queries by 5.03% on the BLEnD benchmark using only self-generated data.
Training a mean-field Transformer under L2 regularization induces an escape from attention-driven token clustering in later layers after initial clustering.
Translation function vectors extracted from one language direction transfer to unseen target languages, indicating a language-agnostic translation signal in multilingual LLMs.
The paper surveys Indic NLP evolution and proposes 'Culture Sensing' to integrate indigenous oral knowledge into foundation models for cultural preservation.
SLO-Guard, a crash-aware two-phase autotuner for vLLM serving, achieves no best-latency improvement over random search but demonstrates more consistent budget allocation across 150 trials on Qwen2-1.5B/A100.
citing papers explorer
-
Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning
Memorized facts in fine-tuned LLMs often sit off the mid-layer reasoning path; relocating those representations recovers most multi-hop generalization failures.
-
Low-Resource Safety Failures Are Action Failures, Not Representation Failures
Low-resource safety failures are action failures because the harmfulness representation transfers but the decision calibration does not; this is fixed by recalibrating a high-resource gate with 1-4 target-language examples.
-
How Far Do Auto-Interpretation Labels Generalize: A Controlled Study Across Languages, Scripts, and Rewordings
Auto-interpretation labels for SAE features generalize poorly across languages and scripts, missing the same semantic content up to 4x more often in Serbian than English and more in Cyrillic than Latin despite deterministic transliteration.
-
Language-Switching Triggers Take a Latent Detour Through Language Models
An 8B autoregressive LM implements a language-switching backdoor via a three-phase circuit with early trigger composition, orthogonal mid-layer propagation, and final-layer MLP conversion, routed through a single-position serial bottleneck.
-
Parallel-SFT: Improving Zero-Shot Cross-Programming-Language Transfer for Code RL
Parallel-SFT mixes parallel programs across languages during SFT to produce more transferable RL initializations, yielding better zero-shot generalization to unseen programming languages.
-
Cross-Lingual Exploration for Parametric Knowledge
Cross-lingual prompt exploration improves factual recall and consistency in LLMs across 17 languages more efficiently than native-language scaling.
-
Want Better Synthetic Data? Steer It: Activation Steering for Low-Resource Language Generation
Activation steering on early layers improves diversity of synthetic data for low-resource languages and often boosts downstream classifier performance compared to non-steered prompting.
-
Large Language Models are Perplexed by some Political Parties
LLMs exhibit higher perplexity on far-right and nationalist party texts than social-democratic ones, consistent across models and languages with correlation to translation metrics.
-
Multilingual Fine-Tuning via Localized Gradient Conflict Resolution
Bucket-Level MOO reformulates multilingual fine-tuning as localized multi-objective optimization and proves it enforces a tighter Pareto stationarity condition while improving cross-lingual performance on four LLMs.
-
Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility
Unlearning in multilingual LLMs suppresses rather than erases knowledge in later layers, with transfer varying by language similarity and reversible via inference-time steering.
-
Multilinguality of Large Language Models From a Structural Perspective
Low-resource languages are structurally more different from English in LLMs than high- or mid-resource ones, and language-specific post-training alters structures while preserving inter-language relationships.
-
DEPART: DEcomposing PARiTy across Multilingual LLMs
A Bayesian framework decomposes mLLM variance, showing language features explain 79-92% of language identity variance and that model identity vs. benchmark-model interactions dominate differently for understanding versus reasoning tasks.
-
Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMs
Four axioms (Causality, Minimality, Separability, Stability) are formalized for latent thought representations; audits of open LLMs on 23 tasks show none satisfy all four and representations add little beyond input embeddings.
-
A quantitative analysis of semantic information in deep representations of text and images
Using an asymmetric rank-based measure, the paper maps which layers of large language and vision models carry shared semantic information, finding central-layer peaks for translations and cross-modal pairs, and systematic predictability asymmetries favoring English and larger models.
-
Parameter Alignment Mitigates Catastrophic Forgetting in Multilingual Expert Language Models
Parameter alignment strategies substantially reduce forgetting in family-based continual pretraining of multilingual LLMs across 32 languages with minimal impact on language acquisition.
-
Cross-Lingual Consensus: Aligning Multilingual Cultural Knowledge via Multilingual Self-Consistency
A multilingual self-consistency plus self-critique method raises cultural alignment scores on English queries by 5.03% on the BLEnD benchmark using only self-generated data.
-
Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers
Training a mean-field Transformer under L2 regularization induces an escape from attention-driven token clustering in later layers after initial clustering.
-
Exploring Language-Agnosticity in Function Vectors: A Case Study in Machine Translation
Translation function vectors extracted from one language direction transfer to unseen target languages, indicating a language-agnostic translation signal in multilingual LLMs.
-
Rethinking Indic AI from a Lens of Cultural Heritage Preservation
The paper surveys Indic NLP evolution and proposes 'Culture Sensing' to integrate indigenous oral knowledge into foundation models for cultural preservation.
-
Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining
SLO-Guard, a crash-aware two-phase autotuner for vLLM serving, achieves no best-latency improvement over random search but demonstrates more consistent budget allocation across 150 trials on Qwen2-1.5B/A100.
- PolyFact: Comparing Consistency-Driven Post-training Methods for Cross-Lingual Factual Recall