REVIEW 4 cited by
TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Learning sentence embeddings often requires a large amount of labeled data. However, for most tasks and domains, labeled data is seldom available and creating it is expensive. In this work, we present a new state-of-the-art unsupervised method based on pre-trained Transformers and Sequential Denoising Auto-Encoder (TSDAE) which outperforms previous approaches by up to 6.4 points. It can achieve up to 93.1% of the performance of in-domain supervised approaches. Further, we show that TSDAE is a strong domain adaptation and pre-training method for sentence embeddings, significantly outperforming other approaches like Masked Language Model. A crucial shortcoming of previous studies is the narrow evaluation: Most work mainly evaluates on the single task of Semantic Textual Similarity (STS), which does not require any domain knowledge. It is unclear if these proposed methods generalize to other domains and tasks. We fill this gap and evaluate TSDAE and other recent approaches on four different datasets from heterogeneous domains.
Forward citations
Cited by 4 Pith papers
-
Deep Learning-based Code Completion: On the Impact on Performance of Contextual Information
A T5-based study of 8 context types for Java code completion finds that combining coding contexts yields +11% relative improvement, and a confidence-based ensemble of context models yields +22%.
-
Mind the Gap: Towards Generalizable Autonomous Penetration Testing via Domain Randomization and Meta-Reinforcement Learning
GAP combines domain randomization via LLM-generated environments with meta-RL to improve generalization of autonomous pentesting agents across unseen host configurations and vulnerabilities.
-
Real-time News Story Identification
A real-time story identification system combining BGE-M3 embeddings, NER, and online clustering attains AMI about 0.57 on Slovene news, well below offline clustering's 0.84.
-
Jasper and Stella: distillation of SOTA embedding models
A 2B-parameter embedding model distilled from two larger teachers achieves a 71.54 average MTEB score (No.3 as of Dec 2024), matching 7B-parameter models.
Discussion (0). Continue with ORCID to comment.