Falcon-180B is a 180B-parameter open decoder-only model trained on 3.5 trillion tokens that approaches PaLM-2-Large performance at lower cost and is released with dataset extracts.
Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
4 Pith papers cite this work. Polarity classification is still indexing.
4
Pith papers citing it
abstract
Recurrent neural networks have been very successful at predicting sequences of words in tasks such as language modeling. However, all such models are based on the conventional classification framework, where the model is trained against one-hot targets, and each word is represented both as an input and as an output in isolation. This causes inefficiencies in learning both in terms of utilizing all of the information and in terms of the number of parameters needed to train. We introduce a novel theoretical framework that facilitates better learning in language modeling, and show that our framework leads to tying together the input embedding and the output projection matrices, greatly reducing the number of trainable variables. Our framework leads to state of the art performance on the Penn Treebank with a variety of network models.
representative citing papers
CTRL is a large conditional transformer language model that uses naturally occurring control codes to steer text generation style and content.
BERT activations show strongest correlation with MEG data for simple sentences; DNN representations generate synthetic brain data that improves stimuli decoding accuracy.
Introduces a multimodal UAV command dataset and shows image-augmented RNN language models outperform text-only versions despite imperfect training associations.
citing papers explorer
-
The Falcon Series of Open Language Models
Falcon-180B is a 180B-parameter open decoder-only model trained on 3.5 trillion tokens that approaches PaLM-2-Large performance at lower cost and is released with dataset extracts.
-
CTRL: A Conditional Transformer Language Model for Controllable Generation
CTRL is a large conditional transformer language model that uses naturally occurring control codes to steer text generation style and content.
-
Relating Simple Sentence Representations in Deep Neural Networks and the Brain
BERT activations show strongest correlation with MEG data for simple sentences; DNN representations generate synthetic brain data that improves stimuli decoding accuracy.
-
Kite: Automatic speech recognition for unmanned aerial vehicles
Introduces a multimodal UAV command dataset and shows image-augmented RNN language models outperform text-only versions despite imperfect training associations.