REVIEW 6 cited by
AnglE-optimized Text Embeddings
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
High-quality text embedding is pivotal in improving semantic textual similarity (STS) tasks, which are crucial components in Large Language Model (LLM) applications. However, a common challenge existing text embedding models face is the problem of vanishing gradients, primarily due to their reliance on the cosine function in the optimization objective, which has saturation zones. To address this issue, this paper proposes a novel angle-optimized text embedding model called AnglE. The core idea of AnglE is to introduce angle optimization in a complex space. This novel approach effectively mitigates the adverse effects of the saturation zone in the cosine function, which can impede gradient and hinder optimization processes. To set up a comprehensive STS evaluation, we experimented on existing short-text STS datasets and a newly collected long-text STS dataset from GitHub Issues. Furthermore, we examine domain-specific STS scenarios with limited labeled data and explore how AnglE works with LLM-annotated data. Extensive experiments were conducted on various tasks including short-text STS, long-text STS, and domain-specific STS tasks. The results show that AnglE outperforms the state-of-the-art (SOTA) STS models that ignore the cosine saturation zone. These findings demonstrate the ability of AnglE to generate high-quality text embeddings and the usefulness of angle optimization in STS.
Forward citations
Cited by 6 Pith papers
-
External Data Extraction Attacks against Retrieval-Augmented Large Language Models
SECRET, an adaptive RAG extraction attack that fuses LLM-optimized jailbreak prompts with cluster-guided queries, extracts large portions of private databases from commercial and open LLMs, including ~35% of a sampled...
-
Beyond Satisfaction: Learning Associations Between Content, Reviews, and Well-Being
Ratings and sentiment loosely track PERMA well-being facets in Goodreads reviews and align more with immediate hedonic than enduring eudaimonic expressions, while book themes like religion and drives correlate with hi...
-
SHIFT: Self-reconstruction Harnesses Implicit Fine-grained Thinking for Retrieval
SHIFT trains LLM retrievers to reason through latent continuous tokens and reconstruct explicit reasoning traces, improving reasoning-intensive retrieval.
-
ExploreGS: Explorable 3D Scene Reconstruction with Virtual Camera Samplings and Diffusion Priors
Adding information-gain-selected virtual views refined by video diffusion priors to 3D Gaussian Splatting improves arbitrary-view rendering quality.
-
Towards Applying Large Language Models to Complement Single-Cell Foundation Models
A fusion model called scMPT, combining scGPT with an LLM text encoder, improves single-cell cell type classification on most tested datasets, and the paper shows the LLM relies on marker genes and simple expression patterns.
-
A Framework for Deductive Semantic Content Analysis at Scale in Science Education Using Text Embeddings
A few-shot text embedding classification framework achieves high agreement with human coders (Cohen's Kappa 0.74-0.83) on a simulated exhaustive coding task over 2,899 physics education survey responses.
Discussion (0). Sign in to comment.