A progressive training scheme with binary-aware initialization and dual-scaling allows pre-trained LLMs to be converted to high-performance 1-bit models without training from scratch.
On the State of the Art of Evaluation in Neural Language Models
5 Pith papers cite this work. Polarity classification is still indexing.
5
Pith papers citing it
abstract
Ongoing innovations in recurrent neural network architectures have provided a steady influx of apparently state-of-the-art results on language modelling benchmarks. However, these have been evaluated using differing code bases and limited computational resources, which represent uncontrolled sources of experimental variation. We reevaluate several popular architectures and regularisation methods with large-scale automatic black-box hyperparameter tuning and arrive at the somewhat surprising conclusion that standard LSTM architectures, when properly regularised, outperform more recent models. We establish a new state of the art on the Penn Treebank and Wikitext-2 corpora, as well as strong baselines on the Hutter Prize dataset.
citation-role summary
background 1
citation-polarity summary
roles
background 1polarities
unclear 1representative citing papers
ChemCrow augments LLMs with 18 expert chemistry tools to autonomously plan and execute syntheses and guide molecular discoveries in organic synthesis, drug discovery, and materials design.
Compressive Transformer sets new records on WikiText-103 (17.1 ppl) and Enwik8 (0.97 bpc) via memory compression and introduces the PG-19 long-range language benchmark.
Systematic evaluation of over 100 ML4H papers finds poorer reproducibility than other ML fields, driven by limited data and code access, and offers recommendations to data providers, publishers, and researchers.
Advocates integrating naturalistic paradigms and AI progress into cognitive science to develop generalizable models of natural behavior while retaining experimental control and theoretical insight.
citing papers explorer
-
Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Models
A progressive training scheme with binary-aware initialization and dual-scaling allows pre-trained LLMs to be converted to high-performance 1-bit models without training from scratch.
-
ChemCrow: Augmenting large-language models with chemistry tools
ChemCrow augments LLMs with 18 expert chemistry tools to autonomously plan and execute syntheses and guide molecular discoveries in organic synthesis, drug discovery, and materials design.
-
Compressive Transformers for Long-Range Sequence Modelling
Compressive Transformer sets new records on WikiText-103 (17.1 ppl) and Enwik8 (0.97 bpc) via memory compression and introduces the PG-19 long-range language benchmark.
-
Reproducibility in Machine Learning for Health
Systematic evaluation of over 100 ML4H papers finds poorer reproducibility than other ML fields, driven by limited data and code access, and offers recommendations to data providers, publishers, and researchers.
-
Naturalistic Computational Cognitive Science: Towards generalizable models and theories that capture the full range of natural behavior
Advocates integrating naturalistic paradigms and AI progress into cognitive science to develop generalizable models of natural behavior while retaining experimental control and theoretical insight.