REVIEW 6 cited by
Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Recent neural network-based language models have benefited greatly from scaling up the size of training datasets and the number of parameters in the models themselves. Scaling can be complicated due to various factors including the need to distribute computation on supercomputer clusters (e.g., TPUs), prevent bottlenecks when infeeding data, and ensure reproducible results. In this work, we present two software libraries that ease these issues: $\texttt{t5x}$ simplifies the process of building and training large language models at scale while maintaining ease of use, and $\texttt{seqio}$ provides a task-based API for simple creation of fast and reproducible training data and evaluation pipelines. These open-source libraries have been used to train models with hundreds of billions of parameters on datasets with multiple terabytes of training data. Along with the libraries, we release configurations and instructions for T5-like encoder-decoder models as well as GPT-like decoder-only architectures. $\texttt{t5x}$ and $\texttt{seqio}$ are open source and available at https://github.com/google-research/t5x and https://github.com/google/seqio, respectively.
Forward citations
Cited by 6 Pith papers
-
On Teacher Hacking in Language Model Distillation
Distilling a student from an imperfect teacher on a fixed offline dataset can make the student match the teacher's errors while drifting from the true target, a phenomenon the authors call teacher hacking.
-
Hymba: A Hybrid-head Architecture for Small Language Models
Hymba-1.5B, a hybrid-head model with parallel attention and Mamba heads plus learnable meta tokens, achieves state-of-the-art accuracy among sub-2B models and outperforms Llama-3.2-3B with an 11.67x smaller cache and ...
-
AnimateAnything: Consistent and Controllable Animation for Video Generation
A two-stage video generation system that converts camera, drag, and reference-video controls into unified optical flows, and adds a frequency-domain stabilizer to reduce flicker.
-
AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment
AmoebaLLM enables instant extraction of arbitrary-shaped subnets from a single fine-tuned LLM, matching or beating dedicated compression methods.
-
Neuro-Symbolic AI in 2024: A Systematic Review
A systematic review of 158 Neuro-Symbolic AI papers finds research concentrated in learning and inference, with explainability, trustworthiness, and Meta-Cognition as underrepresented gaps.
-
Building Task Bots with Self-learning for Enhanced Adaptability, Extensibility, and Factuality
A thesis that combines self-learning from dialog logs, schema-guided prompting, and self-aligned factuality to build task bots with minimal human intervention.
Discussion (0). Continue with ORCID to comment.