Auditability of subliminal learning is constrained by channel location, with initialization-dependent body channels allowing pre-training screens while vocabulary geometry and conditional body channels evade them.
hub
How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings
17 Pith papers cite this work, alongside 9 external citations. Polarity classification is still indexing.
17
Pith papers citing it
9
external citations · Pith
abstract
Replacing static word embeddings with contextualized word representations has yielded significant improvements on many NLP tasks. However, just how contextual are the contextualized representations produced by models such as ELMo and BERT? Are there infinitely many context-specific representations for each word, or are words essentially assigned one of a finite number of word-sense representations? For one, we find that the contextualized representations of all words are not isotropic in any layer of the contextualizing model. While representations of the same word in different contexts still have a greater cosine similarity than those of two different words, this self-similarity is much lower in upper layers. This suggests that upper layers of contextualizing models produce more context-specific representations, much like how upper layers of LSTMs produce more task-specific representations. In all layers of ELMo, BERT, and GPT-2, on average, less than 5% of the variance in a word's contextualized representations can be explained by a static embedding for that word, providing some justification for the success of contextualized representations.
hub tools
citation-role summary
background 4
citation-polarity summary
years
2026 17roles
background 4representative citing papers
ResRL decouples shared semantics between positive and negative responses in LLM reinforcement learning via SVD-based projection residuals, outperforming baselines including NSR by up to 9.4% on math reasoning benchmarks.
mEOL creates aligned embeddings for text, images, and SVGs using instruction-guided MLLM one-word summaries and semantic SVG rewriting, outperforming baselines on a new text-to-SVG retrieval benchmark.
Aggregating per-token pullback metrics via the Fréchet mean on the SPD manifold outperforms Euclidean mean pooling for sentence classification, with most of the gain attributable to geometric aggregation rather than learned encoder structure.
Presents a diagnostic framework for semantic ID tokenizer failures using overlap and capacity metrics and proposes DRQ to separate geometry from distribution matching.
NITP augments next-token prediction with cosine alignment to stop-gradient shallow-layer features of the next token, improving geometry and downstream scores at ~2% extra training FLOPs.
HyperPersona is a hypergraph framework that jointly models document, sentence, and word levels of text via hyperedges and nodes, then uses a transformer graph encoder to predict Big Five personality traits from text alone.
REALISTA generates semantically coherent adversarial prompts via latent-space optimization over input-dependent editing directions, achieving stronger hallucination elicitation than prior realistic attacks on open-source and reasoning LLMs.
Geometry-calibrated conformal abstention lets language models abstain from uncertain queries with finite-sample guarantees on both participation rate and conditional correctness of answers.
REZE controls representation shifts in contrastive pre-finetuning of text embeddings via eigenspace decomposition of anchor-positive pairs and adaptive soft-shrinkage on task-variant directions.
GPC learns a motion vocabulary via Finite Scalar Quantization and end-to-end RL, then trains an autoregressive transformer for next-token control generation, achieving 99.98% motion reproduction success with emergent robustness.
A mean-field analysis of embedding-space crowding shows a phase transition and Fokker-Planck dynamics that drive retrieval-augmented agents to self-organize toward exclusive service of majority interests.
HybridMoE with controlled hybridization and idiomatic property signals yields 5-6% gains in figurative language representation for multilingual vision-language models.
Learned anchors as semantic prototypes combined with whitened inner products improve relative representations, enabling nearly lossless zero-shot communication between heterogeneous neural models on vision and language tasks.
An empirical audit of 22 JEPA-style training auxiliaries on Llama-3.2-1B fine-tuning for regex generation finds no statistically significant task improvement after multiple-testing correction, even when auxiliaries visibly alter hidden-state geometry.
HyperLens reveals that deeper transformer layers magnify small confidence changes into fine-grained trajectories, allowing quantification of cognitive effort where complex tasks demand more and standard SFT can reduce it.
A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.
citing papers explorer
-
Channel Location Constrains the Auditability of Subliminal Learning
Auditability of subliminal learning is constrained by channel location, with initialization-dependent body channels allowing pre-training screens while vocabulary geometry and conditional body channels evade them.
-
ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning
ResRL decouples shared semantics between positive and negative responses in LLM reinforcement learning via SVD-based projection residuals, outperforming baselines including NSR by up to 9.4% on math reasoning benchmarks.
-
mEOL: Training-Free Instruction-Guided Multimodal Embedder for Vector Graphics and Image Retrieval
mEOL creates aligned embeddings for text, images, and SVGs using instruction-guided MLLM one-word summaries and semantic SVG rewriting, outperforming baselines on a new text-to-SVG retrieval benchmark.
-
Riemannian Geometry for Pre-trained Language Model Embeddings
Aggregating per-token pullback metrics via the Fréchet mean on the SPD manifold outperforms Euclidean mean pooling for sentence classification, with most of the gain attributable to geometric aggregation rather than learned encoder structure.
-
Decoupled Residual Quantization for Robust Semantic IDs in Recommendation
Presents a diagnostic framework for semantic ID tokenizer failures using overlap and capacity metrics and proposes DRQ to separate geometry from distribution matching.
-
NITP: Next Implicit Token Prediction for LLM Pre-training
NITP augments next-token prediction with cosine alignment to stop-gradient shallow-layer features of the next token, improving geometry and downstream scores at ~2% extra training FLOPs.
-
HyperPersona: A Multi-Level Hypergraph Framework for Text-Based Automatic Personality Prediction
HyperPersona is a hypergraph framework that jointly models document, sentence, and word levels of text via hyperedges and nodes, then uses a transformer graph encoder to predict Big Five personality traits from text alone.
-
REALISTA: Realistic Latent Adversarial Attacks that Elicit LLM Hallucinations
REALISTA generates semantically coherent adversarial prompts via latent-space optimization over input-dependent editing directions, achieving stronger hallucination elicitation than prior realistic attacks on open-source and reasoning LLMs.
-
Geometry-Calibrated Conformal Abstention for Language Models
Geometry-calibrated conformal abstention lets language models abstain from uncertain queries with finite-sample guarantees on both participation rate and conditional correctness of answers.
-
REZE: Representation Regularization for Domain-adaptive Text Embedding Pre-finetuning
REZE controls representation shifts in contrastive pre-finetuning of text embeddings via eigenspace decomposition of anchor-positive pairs and adaptive soft-shrinkage on task-variant directions.
-
GPC: Large-Scale Generative Pretraining for Transferable Motor Control
GPC learns a motion vocabulary via Finite Scalar Quantization and end-to-end RL, then trains an autoregressive transformer for next-token control generation, achieving 99.98% motion reproduction success with emergent robustness.
-
The Crowded Embedding Space: A Mean-Field Mechanism for Emergent Marginalization in Retrieval-Augmented Agents
A mean-field analysis of embedding-space crowding shows a phase transition and Fokker-Planck dynamics that drive retrieval-augmented agents to self-organize toward exclusive service of majority interests.
-
When Meaning Travels: A Granular Lens on Hybrid-MoE's Role in Idiomatic Understanding for Language Models
HybridMoE with controlled hybridization and idiomatic property signals yields 5-6% gains in figurative language representation for multilingual vision-language models.
-
Improving Relative Representations with Learned Anchors and Whitened Inner Products
Learned anchors as semantic prototypes combined with whitened inner products improve relative representations, enabling nearly lossless zero-shot communication between heterogeneous neural models on vision and language tasks.
-
Representation Without Reward: A JEPA Audit for LLM Fine-Tuning
An empirical audit of 22 JEPA-style training auxiliaries on Llama-3.2-1B fine-tuning for regex generation finds no statistically significant task improvement after multiple-testing correction, even when auxiliaries visibly alter hidden-state geometry.
-
HyperLens: Quantifying Cognitive Effort in LLMs with Fine-grained Confidence Trajectory
HyperLens reveals that deeper transformer layers magnify small confidence changes into fine-grained trajectories, allowing quantification of cognitive effort where complex tasks demand more and standard SFT can reduce it.
-
The Hitchhiker's Guide to Agentic AI: From Foundations to Systems
A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.