RECORD
Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks
Machine Learning · Submitted 31 Oct 2023
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim We demonstrate that LC-PFN can approximate the posterior predictive distribution more accurately than MCMC, while being over 10 000 times faster.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Learning curve extrapolation aims to predict model performance in later epochs of training, based on the performance in earlier epochs. In this work, we argue that, while the inherent uncertainty in the extrapolation of learning curves warrants a Bayesian approach, existing methods are (i) overly restrictive, and/or (ii) computationally expensive. We describe the first application of prior-data fitted neural networks (PFNs) in this context. A PFN is a transformer, pre-trained on data generated from a prior, to perform approximate Bayesian inference in a single forward pass. We propose LC-PFN, a PFN trained to extrapolate 10 million artificial right-censored learning curves generated from a parametric prior proposed in prior art using MCMC. We demonstrate that LC-PFN can approximate the posterior predictive distribution more accurately than MCMC, while being over 10 000 times faster. We also show that the same LC-PFN achieves competitive performance extrapolating a total of 20 000 real learning curves from four learning curve benchmarks (LCBench, NAS-Bench-201, Taskset, and PD1) that stem from training a wide range of model architectures (MLPs, CNNs, RNNs, and Transformers) on 53 different datasets with varying input modalities (tabular, image, text, and protein data). Finally, we investigate its potential in the context of model selection and find that a simple LC-PFN based predictive early stopping criterion obtains 2 - 6x speed-ups on 45 of these datasets, at virtually no overhead.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Learning curve extrapolation aims to predict model performance in later epochs of training, based on the performance in earlier epochs. | background 1.00 | 0.04 | 0.02 | asserted 0.99 |
| s1 | In this work, we argue that, while the inherent uncertainty in the extrapolation of learning curves warrants a Bayesian approach, existing methods are (i) overly restrictive, and/or (ii) computationally expensive. | gap 0.98 | 0.20 | 0.11 | asserted 0.96 |
| s2 | We describe the first application of prior-data fitted neural networks (PFNs) in this context. | contribution 1.00 | 0.91 | 0.07 | asserted 1.00 |
| s3 | A PFN is a transformer, pre-trained on data generated from a prior, to perform approximate Bayesian inference in a single forward pass. | background 0.85 | 0.09 | 0.04 | asserted 1.00 |
| s4 | We propose LC-PFN, a PFN trained to extrapolate 10 million artificial right-censored learning curves generated from a parametric prior proposed in prior art using MCMC. | method 0.90 | 0.92 | 0.07 | asserted 0.98 |
| s5 | We demonstrate that LC-PFN can approximate the posterior predictive distribution more accurately than MCMC, while being over 10 000 times faster. | result 1.00 | 0.96 | 0.45 | asserted 1.00 |
| s6 | We also show that the same LC-PFN achieves competitive performance extrapolating a total of 20 000 real learning curves from four learning curve benchmarks (LCBench, NAS-Bench-201, Taskset, and PD1) that stem from training a wide range of model architectures (MLPs, CNNs, RNNs, and Transformers) on 53 different datasets with varying input modalities (tabular, image, text, and protein data). | result 0.99 | 0.95 | 0.23 | asserted 1.00 |
| s7 | Finally, we investigate its potential in the context of model selection and find that a simple LC-PFN based predictive early stopping criterion obtains 2 - 6x speed-ups on 45 of these datasets, at virtually no overhead. | result 0.95 | 0.96 | 0.18 | asserted 0.99 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Learning curve extrapolation aims to predict model performance in later epochs of training, based on the performance in earlier epochs. In this work, we argue that, while the inherent uncertainty in the extrapolation of learning curves warrants a Bayesian approach, existing methods are (i) overly restrictive, and/or (ii) computationally expensive. We describe the first application of prior-data fitted neural networks (PFNs) in this context. A PFN is a transformer, pre-trained on data generated from a prior, to perform approximate Bayesian inference in a single forward pass. We propose LC-PFN, a PFN trained to extrapolate 10 million artificial right-censored learning curves generated from a parametric prior proposed in prior art using MCMC. We demonstrate that LC-PFN can approximate the posterior predictive distribution more accurately than MCMC, while being over 10 000 times faster. We also show that the same LC-PFN achieves competitive performance extrapolating a total of 20 000 real learning curves from four learning curve benchmarks (LCBench, NAS-Bench-201, Taskset, and PD1) that stem from training a wide range of model architectures (MLPs, CNNs, RNNs, and Transformers) on 53 different datasets with varying input modalities (tabular, image, text, and protein data). Finally, we investigate its potential in the context of model selection and find that a simple LC-PFN based predictive early stopping criterion obtains 2 - 6x speed-ups on 45 of these datasets, at virtually no overhead.
Human review
References
How the literature uses it
2 arXiv papers cite this in 6 sentences
Papers citing it · 4 sentences without a confident label
Compares with it 1 paper
-
A recent approach that reports results comparable to or better than the MCMC approach of [ref] was presented in [ref] .
The Unreasonable Effectiveness Of Early Discarding After One Epoch In Neural Network Hyperparameter Optimization · Extrapolation via Prior Fitted Networks (PFN) · 0.81
Cites it as background 1 paper
-
They either assume a parametric model [ref] , Gaussian process model [ref] , complex hierarchical Bayesian models [ref] , or Bayesian neural network models [ref] to model the learning curves, to name a few examples.
The Unreasonable Effectiveness Of Early Discarding After One Epoch In Neural Network Hyperparameter Optimization · Related Work · 0.94
What it cites · 9 sentences without a confident label
Builds on 1 paper
-
In particular, while [ref] only considered curves of accuracy over epochs from LCBench [ref] , we generalize the approach to also support curves of logloss from Taskset [ref] , and over training sample size from LCDB [ref] ; * describing with a novel data-driven prior that is better capable of modeling the at times erratic nature of real-world learning curves than traditional parametric priors than, e.g., the one by [ref] ; and * and demonstrate its applicability in the context of automated model selection.
Using a thousand optimization tasks to learn hyperparameter search strategies · Related work · 1.00
Uses data or values from 1 paper
-
Our dataset comprises 20 000 learning curves, sourced from four distinct benchmarks: [ref] , [ref] , [ref] and [ref] , each contributing 5 000 curves, randomly selected from specific subtasks.
Using a thousand optimization tasks to learn hyperparameter search strategies · Extrapolating real-world learning curves · 1.00
Cites as background 2 papers
-
This setting is also known as vertical model selection [ref] and was also considered by [ref] .
Learning Curves for Decision Making in Supervised Machine Learning: A Survey · Application: Extrapolation-based early stopping in model selection · 0.97
-
For example, the Freeze-Thaw Bayesian optimization method [ref] used a Gaussian process (GP) as a joint model of learning curves and hyperparameters to decide what learning run to continue for a few epochs (or whether to start a new one).
Freeze-Thaw Bayesian Optimization · Related work · 0.96
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.