REVIEW 19 cited by
Whitening Sentence Representations for Better Semantics and Faster Retrieval
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Whitening Sentence Representations for Better Semantics and Faster Retrieval
read the original abstract
Pre-training models such as BERT have achieved great success in many natural language processing tasks. However, how to obtain better sentence representation through these pre-training models is still worthy to exploit. Previous work has shown that the anisotropy problem is an critical bottleneck for BERT-based sentence representation which hinders the model to fully utilize the underlying semantic features. Therefore, some attempts of boosting the isotropy of sentence distribution, such as flow-based model, have been applied to sentence representations and achieved some improvement. In this paper, we find that the whitening operation in traditional machine learning can similarly enhance the isotropy of sentence representations and achieve competitive results. Furthermore, the whitening technique is also capable of reducing the dimensionality of the sentence representation. Our experimental results show that it can not only achieve promising performance but also significantly reduce the storage cost and accelerate the model retrieval speed.
Forward citations
Cited by 19 Pith papers
-
SimCSE: Simple Contrastive Learning of Sentence Embeddings
SimCSE achieves 76.3% unsupervised and 81.6% supervised Spearman's correlation on STS tasks with BERT-base, improving prior best results by 4.2% and 2.2% via simple contrastive learning.
-
Anisotropy Decides Cosine vs. Rank Metrics for Text Embeddings
Anisotropy, quantified by dominant-dimension variance fraction, determines the best parameter-free similarity metric for text embeddings, with rank-based metrics gaining ~20% relative where cosine is weakest.
-
CrossAlpha: An Annual-Report Benchmark for Cross-Market Factor Researc (with LLM Agents)
CrossAlpha is a new open benchmark covering 3600 firms across US, Japan, Taiwan, South Korea and Hong Kong that distills reports into 10-category descriptions, constructs residual cross-market similarity graphs, and t...
-
Concepts Whisper While Syntax Shouts: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations
Transformer activations show spectral anti-concentration for concepts in the tail while syntax prefers high-variance directions, forming a dual geometry.
-
Spectral Tempering for Embedding Compression in Dense Passage Retrieval
Spectral Tempering derives an adaptive scaling factor γ(k) from the embedding eigenspectrum via local SNR analysis and knee-point normalization to achieve near-optimal compression without training or validation.
-
Recovering Latent Structures after Variational Bayesian Variable Selection: Fit Assessment and Factor-Number Selection in Partially Exploratory Factor Analysis
A scale-free gain rule applied to variational ELBO paths recovers true factor dimensionality in partially exploratory factor analysis where raw information criteria over-factor.
-
Semantic Code Clone Detection: Are We There Yet?
SOTA semantic code clone detectors exhibit substantial performance degradation on distribution-shifted yet semantically equivalent clones, revealing reliance on lexical and structural shortcuts rather than semantic un...
-
When Global Gating Is Enough: Admission-Time Hubness Control in Anisotropic Vector Retrieval Systems
Global admission-time gating with sentinel queries controls vector hubness in anisotropic embeddings, achieving recall 1.0 on critical attack points and 0.91 on HotFlip attacks with 1% false positives, while per-topic...
-
Semantic Identification of IoT Devices from Behavioral Primitives
Semantic matching of MUD ACE behavioral primitives yields more robust IoT device identification than exact overlap under runtime variations and sparse observations.
-
Your UnEmbedding Matrix is Secretly a Feature Lens for Text Embeddings
EmbedFilter applies a linear filter derived from the LLM unembedding matrix to suppress high-frequency token influences in text embeddings, yielding improved zero-shot performance and inherent dimensionality reduction.
-
Context Memorization for Efficient Long Context Generation
Attention-state memory externalizes long prefixes into a lightweight lookup table of precomputed attention states, yielding higher accuracy than standard in-context learning at fixed memory budgets and lower latency t...
-
ASPIRE: Make Spectral Graph Collaborative Filtering Great Again via Adaptive Filter Learning
ASPIRE learns adaptive graph filters via bi-level optimization to overcome low-frequency explosion bias in spectral collaborative filtering, achieving strong performance and stability.
-
REZE: Representation Regularization for Domain-adaptive Text Embedding Pre-finetuning
REZE controls representation shifts in contrastive pre-finetuning of text embeddings via eigenspace decomposition of anchor-positive pairs and adaptive soft-shrinkage on task-variant directions.
-
Text and Code Embeddings by Contrastive Pre-Training
Contrastive pre-training on unsupervised data at scale creates text and code embeddings that set new state-of-the-art results on classification and semantic search benchmarks.
-
Correlation Is Not Enough: Embedding Human Metadata for Individual Causal Discovery
Contrastive fine-tuning on 72k pairs plus BODHI hard-negative mining from a biomedical KG improves PubMedBERT separation from 1.05x to 2.30x and cross-domain discrimination by +0.392 while preserving most BIOSSES performance.
-
Towards General Text Embeddings with Multi-stage Contrastive Learning
GTE_base is a compact text embedding model using multi-stage contrastive learning on diverse data that outperforms OpenAI's API and 10x larger models on massive benchmarks and works for code as text.
-
MODE-RAG: Manifold Outlier Diagnosis and Energy-based Retrieval-Augmented Generation Evaluation
MODE-RAG introduces a VFE-driven multi-agent pipeline with MCTS and logit perturbations to lower hallucination and sycophancy rates in multimodal RAG, tested on the new ModeVent subset of MultiVent.
-
The Hitchhiker's Guide to Agentic AI: From Foundations to Systems
A comprehensive reference book organizing existing techniques for agentic AI systems across LLM substrate, reasoning, agent design patterns, inter-agent coordination, and production deployment.
-
The Hitchhiker's Guide to Agentic AI: From Foundations to Systems
A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.