REVIEW 5 cited by
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Transformer-based pretrained language models (T-PTLMs) have achieved great success in almost every NLP task. The evolution of these models started with GPT and BERT. These models are built on the top of transformers, self-supervised learning and transfer learning. Transformed-based PTLMs learn universal language representations from large volumes of text data using self-supervised learning and transfer this knowledge to downstream tasks. These models provide good background knowledge to downstream tasks which avoids training of downstream models from scratch. In this comprehensive survey paper, we initially give a brief overview of self-supervised learning. Next, we explain various core concepts like pretraining, pretraining methods, pretraining tasks, embeddings and downstream adaptation methods. Next, we present a new taxonomy of T-PTLMs and then give brief overview of various benchmarks including both intrinsic and extrinsic. We present a summary of various useful libraries to work with T-PTLMs. Finally, we highlight some of the future research directions which will further improve these models. We strongly believe that this comprehensive survey paper will serve as a good reference to learn the core concepts as well as to stay updated with the recent happenings in T-PTLMs.
Forward citations
Cited by 5 Pith papers
-
Semantic Homogenization in Italian Popular Music: A Diachronic Analysis
Sanremo lyrics exhibit rising semantic homogeneity over decades, consistently recovered by full-text, portion, topic and word-level embedding analyses.
-
A Hybrid Framework for Subject Analysis: Integrating Embedding-Based Regression Models with Large Language Models
Using an ML-predicted label count to constrain LLM generation and post-editing outputs to the LCSH vocabulary lifts subject-heading prediction F1 from 0.135 to 0.300 on a 2,100-book test set.
-
Few-shot Learning on AMS Circuits and Its Application to Parasitic Capacitance Prediction
A few-shot graph-pretraining pipeline, built from subgraph sampling and a hybrid graph transformer, predicts parasitic coupling capacitance on unseen AMS circuits with substantially lower error than prior graph baselines.
-
Brain Network Analysis Based on Fine-tuned Self-supervised Model for Brain Disease Diagnosis
An MLP adapter before a frozen pre-trained brain transformer improves AD vs NC and MCI vs NC classification on ADNI relative to BrainNetCNN and BrainGNN, but the gain over the transformer alone is untested.
-
Synthetic Time Series Forecasting with Transformer Architectures: Extensive Simulation Benchmarks
Autoformer and PatchTST outperform Informer across synthetic forecasting benchmarks, while a proposed Koopman-Transformer hybrid is illustrated on Van der Pol and Lorenz systems.
Discussion (0). Sign in to comment.