{"id":"19056064-dff8-4064-8794-a6e3b02fba89","arxiv_id":"2505.23032","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A Prior-data Fitted Network with a scaling-law-specific prior gives better point and uncertainty predictions for neural scaling law extrapolation than MCMC, BNSL, and LC-PFN baselines.","lead":"This paper trains a Transformer to extrapolate neural scaling laws in a Bayesian way, using a synthetic prior that covers power laws, breaks, and double-descent shapes. It reports better error and uncertainty estimates than MCMC and previous learning-curve methods on public benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is load-bearing on the functional prior's coverage: Section 4.3 tuned the prior to visually match and optimize against the evaluation curves, so the reported gains may not transfer to new scaling-law families.","rationale":"The paper is a solid engineering contribution: it releases code, defines a clear prior, and evaluates on extensive datasets; the PFN framework is a sensible way to amortize Bayesian extrapolation, and the ablation studies and efficiency analysis are useful. However, the core scientific claim of 'superior performance' and 'reliable uncertainty' depends on the prior matching the deployment distribution. Because the prior hyperparameters were tuned on the evaluation datasets—both manually in Section 4.3 and via BO on the test-average RMSLE in Appendix D—the evaluation is not independent. This is exactly the reader's weakest assumption, and I agree with it. The concern is tractable: an out-of-prior evaluation either confirms transferability or exposes the boundary of the method's applicability. My read does not change the reader's conditional verdict; it reinforces it. No internal inconsistency or author misconduct is indicated; the paper is transparent about its prior tuning, which makes the proposed test feasible.","tokens_in":37857,"tokens_out":9102,"duration_ms":103375,"concrete_test":"Evaluate the released NSL-PFN, with no retuning of the prior, on a hold-out set of scaling-law curves that were not part of the prior-design loop. A decisive family is a synthetic but realistic one outside the three template families: e.g., an upward-first curve (BetaCDF-like upward segment followed by a downward M4 segment, 'Up-Down') or a power law with a smooth plateau. Alternatively, use real scaling laws from Hoffmann et al. (2022) or another recent LLM study not in IC/NLP/Nano/ColPret/DD. Compare RMSLE and log-likelihood against MCMC (BNSL) with validation-based break selection and LC-PFN. If NSL-PFN does not retain its advantage, the central claim is limited to the prior's tuned support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The method is an amortized Bayesian inference over a hand-designed functional prior (Section 3.1). The PFN's posterior predictive quality is entirely determined by this prior, and Section 4.3 states the prior parameters were 'manually adjusted to visually match the shapes of the actual curves' collected from the evaluation datasets, followed by Bayesian optimization on the average RMSLE of the same benchmarks (Figure 7, Table 13). The evaluation therefore does not test extrapolation to new scaling-law families; it tests a prior that was fit to the test shapes. The abstract's claim of 'superior performance' is only supported for curves within the three template families (Down, Down-Down, Down-Up-Down) whose prior hyperparameters were tuned on the test data. Additionally, the cutoff restrictions in Section 3.1 exclude contexts that end before a break in downward-trending curves, meaning the model is untrained for some data-limited scenarios later evaluated on the DD dataset (e.g., 40% cutoff before the upward segment). If a practitioner applies NSL-PFN to a new architecture or dataset whose scaling behavior falls outside this prior support, nothing in the paper guarantees the reported RMSLE/LL advantage or uncertainty calibration will persist.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NSL-PFN, a Prior-data Fitted Network for Bayesian neural scaling law extrapolation. The authors design a functional prior over scaling-law curves by concatenating M3/M4 power-law segments with random breaks and occasional upward BetaCDF segments, then train a Transformer to output posterior predictive distributions. The method is evaluated on image classification (IC), NLP, nanoGPT-Bench (Nano), ColPret, and double-descent (DD) datasets against M1--M4 and BNSL point estimators, MCMC variants of these models, LC-PFN, BLR, and DKGP. The central claims are that NSL-PFN gives better point predictions (RMSLE) and better uncertainty quantification (log-likelihood and MSCE) than the baselines, especially in data-limited settings, and that its amortized inference is far cheaper than MCMC.","tokens_in":38103,"tokens_out":7508,"duration_ms":81200,"significance":"If the central claims hold, this is a practically useful contribution: it provides fast, uncertainty-aware extrapolation for neural scaling laws, with code and models released. The empirical effort is substantial: six benchmark suites, calibration metrics, active-learning experiments, ablations of the prior components, and an analysis of MCMC sample counts. The main reservation is that the functional prior was tuned on the evaluation data themselves, so the reported gains may largely reflect prior coverage of the test class rather than general extrapolation ability. Because the PFN is entirely defined by its prior, this is a load-bearing concern for the paper's main claim.","major_comments":[{"comment":"The evaluation protocol is circular for the paper's central claim. Section 4.3 states that the prior hyperparameters in Table 1 were 'manually adjusted to visually match the shapes of the actual curves collected from various domains' and then further optimized by Bayesian optimization on the average RMSLE of the same benchmarks (Fig. 7, Table 13). Since the PFN is trained exclusively on samples from this prior, using the evaluation curves to set the prior means the results in Tables 2-4, Fig. 3, and Table 9 partly measure how well the prior was fit to the test class. Please add a genuinely out-of-prior evaluation: fix the prior without inspecting the evaluation curves, or hold out entire scaling-law families or datasets during prior tuning, and report RMSLE, LL, and MSCE on those held-out curves. Without this, the abstract's 'superior performance' claim is not supportable beyond the tuned benchmark suite.","section":"§4.3, Table 1, Fig. 7"},{"comment":"The cutoff restrictions exclude from training exactly the data-limited scenarios that are later evaluated. Down-Down curves only allow cutoffs within the last segment, and Down-Up-Down curves only allow cutoffs in the second or third segment; however, the DD evaluations in Fig. 4 and Figs. 16-23 include 40% cutoffs in which the context lies entirely in the first decreasing segment before the upward break. For such inputs the model has never seen a valid training context of that shape, so the reported DD results at low cutoffs are out-of-support predictions. Please either train with cutoffs before the first break, or explicitly restrict and clearly label the DD evaluation to contexts that are within the training cutoff support. The 'inherently difficult' argument in §4.1 does not resolve the mismatch between the training distribution and the evaluation distribution.","section":"§3.1 (cutoff distribution) and §4.1/Fig. 4"},{"comment":"The BNSL baseline is not evaluated with the method as described by Eq. (5). The implementation in Eq. (11) is a neural-network reparameterization, and the number of breaks and validation procedure are a reimplementation based on the paper description, with four variants compared in Table 10. This means the BNSL and MCMC(BNSL) results in Tables 2-4 may not reflect the actual BNSL method. Please use the official implementation if one is available; otherwise, report which variant was used and justify why it is the fairest, and provide failure counts for the excluded trials.","section":"§C, Eq. (11) and Table 10"},{"comment":"The MCMC baselines use priors that appear poorly calibrated and possibly invalid, e.g., loga ~ U(1,1000) for MCMC(M4) and a ~ U(-1,0.5) for MCMC(M1) despite positive-valued targets. Such priors can produce extremely diffuse or invalid proposals and may handicap the Bayesian baselines. This is important because the paper's claim of superiority over Bayesian baselines rests on these comparisons. Please report results with more standard, clearly justified priors, or include a sensitivity analysis over prior choices.","section":"§C, Table 11"}],"minor_comments":[{"comment":"The footnote '† indicates that some trials failed due to overflow error, which were excluded from the calculation' should specify how many trials failed, for which methods and datasets, and whether the reported averages and standard deviations are over the remaining trials only.","section":"Table 4"},{"comment":"The interpolation-loss training procedure is described only briefly ('randomly sample a subset of T to add them to C'). Please specify the distribution over the number or fraction of target points moved to the context, as this affects the active-learning model and reproducibility.","section":"§3.2 and §4.2"},{"comment":"The caption states that convergence failures were observed for some MCMC(BNSL) runs and that these runs were excluded; please report the number of excluded runs at each sample count, since selective exclusion can bias the comparison.","section":"Fig. 6"},{"comment":"The MCMC priors are not matched across baselines: for example, MCMC(M4) uses a different x-normalization (maximum 1000) from the other MCMC variants. Please justify this asymmetry or use a common normalization.","section":"§C, Table 11"}],"recommendation":"major_revision","confidential_remarks":"The main issue is the evaluation protocol, not the methodology itself. The paper contains substantial empirical work and a plausible method, but the prior was tuned on the evaluation curves, and the DD evaluation includes contexts outside the training cutoff support. These issues are fixable within the scope of a revision, so I do not recommend rejection. I see no evidence of fabrication; the concerns are about protocol and baseline fairness."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something real: it builds a Prior-data Fitted Network whose prior explicitly covers power laws, breaks, and double-descent shapes, then tests it on a wide range of scaling-law benchmarks. The prior design is thoughtful, the ablations show each component helps, and the code release plus the efficiency numbers make it a practical candidate for people who need quick uncertainty-aware extrapolation. I give it credit for shipping a working system.\n\nThe soft spots are real and land on the main claim. Section 4.3 says the prior parameters were manually adjusted to visually match the shapes of the actual curves from the evaluation datasets, then further tuned with Bayesian optimization on the average RMSLE of those same benchmarks. That means the reported RMSLE, log-likelihood, and calibration numbers measure how well the prior covers the test class, not how well the method generalizes to a new scaling-law family. The stress-test note is right about this being load-bearing: the prior is the method, so tuning it on the test data inflates every headline number.\n\nThe baseline protocol has related issues. The BNSL baseline is a reimplementation, and the variant used in the main text (last-10% validation, no cropping) was chosen after comparing options on the evaluation datasets. The MCMC baselines use only 150 samples and generic priors; the paper shows more samples give marginal gains, but that experiment uses the same test data and does not fully redeem the comparison. There is also a mismatch between training and evaluation: the cutoff restriction in Section 3.1 means the model never sees a double-decent context that ends in the first downward segment, yet the DD evaluation includes those cases. The paper labels one such case inherently difficult but still averages over it.\n\nOne smaller point: the claim of being the first Bayesian method for neural scaling law extrapolation is overstated, since MCMC on M1-M4 and BNSL is already Bayesian. The first use of PFNs specifically is a fairer contribution.\n\nWho should read this? People working on scaling-law prediction or in-context Bayesian inference will get a well-engineered method and a good reminder of how easy it is to make a prior look good on benchmarks you tuned it to. It deserves a serious referee, but the revision should require an out-of-prior evaluation, a fairer baseline selection protocol, and claims that match what was actually shown.","headline":"A useful PFN application to scaling-law extrapolation, but prior tuning on the test benchmarks keeps the headline claims from being fully established.","tokens_in":38666,"tokens_out":2348,"would_cite":false,"duration_ms":27264,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a Prior-data Fitted Network meta-trained on synthetic power-law curves outperforms MCMC and LC-PFN baselines at neural scaling law extrapolation, with better-calibrated uncertainty and improved Bayesian active…","keywords":["neural scaling laws","Bayesian inference","prior-data fitted networks","uncertainty quantification","power law extrapolation","double descent","in-context learning","Bayesian active learning"],"falsifier":"Generate a family of scaling curves that is structurally absent from the Section 3.1 prior—for example, an exponential saturation curve with a long flat plateau at both ends, or a curve that keeps rising after a downward segment—and compare NSL-PFN against MCMC(BNSL) on RMSLE and MSCE; if NSL-PFN loses calibration or accuracy on these out-of-prior shapes, the prior-support assumption marks the limits of the claim.","tokens_in":37631,"feed_emoji":"📈","tokens_out":8783,"duration_ms":81660,"temperature":0.7,"pith_summary":"The paper sets out to show that extrapolating neural scaling laws—predicting how loss or error continues as data, model size, or compute grows—is best treated as a Bayesian inference problem, and that a Prior-data Fitted Network (a Transformer that meta-learns to output a posterior predictive distribution) can solve it better than fitting parametric curves by point estimation or MCMC. The authors design a functional prior over synthetic scaling curves made of power-law segments with randomly occurring breaks and occasional upward segments, then train the network on 1.6 million sampled curves. They claim NSL-PFN beats MCMC versions of M1–M4 and BNSL, the LC-PFN, and generic Bayesian regressors on RMSLE and log-likelihood across image, language, ColPret, and double-descent datasets, with better-calibrated uncertainty and better Bayesian active learning performance in data-limited settings. If this holds, compute-budget decisions—whether to spend more resources for an expected gain—can be made with quantified reliability rather than a single point forecast.","feed_headline":"Bayesian meta-learned model beats MCMC at scaling-law extrapolation","feed_subtitle":"A transformer trained on synthetic scaling curves predicts performance tails with calibrated uncertainty.","key_machinery":"The load-bearing object is the neural scaling law prior: a hierarchical generative process in which each curve is stitched from segments. Downward segments are drawn from M3, $y=a(x-1+d)^b$, or M4, whose inverse form is $x = (y/(a(1-y)^{\\alpha}))^{-1/b}$ with $c=0$ and $y_0=1$; upward segments use a Beta CDF with an S-shape; the number and locations of breaks are sampled randomly; each segment is renormalized and noise is added. A cutoff distribution then fixes which part of the curve is context and which is target, deliberately excluding cutoffs after which a surprise break is judged unpredictable. Training the PFN on 1.6 million such synthetic curves with an autoregressive context loss plus an interpolation variant gives the network the inductive bias to extrapolate power-law tails, infer breaks automatically, and quantify uncertainty without closed-form posteriors.","core_discovery":"The central claim, stated on the paper's own terms, is that a meta-learned in-context Bayesian model with a scaling-law-shaped prior outperforms both classical parametric point estimates and the natural MCMC baselines at predicting the unseen tail of neural scaling curves. The method, NSL-PFN, ingests the observed portion of a curve as context and returns a full predictive distribution at extrapolated inputs, so it simultaneously supplies a point fit (the median), a functional form, an inferred number of breaks, and calibrated uncertainty. Empirically the paper reports the best average RMSLE and log-likelihood among all compared methods on the image classification benchmark, the NLP/Nano suite, and the ColPret and double-descent datasets, and the best MSCE calibration on ColPret and DD. The authors further claim that these uncertainties drive effective Bayesian active learning, selecting the next observation point by variation ratio, and that the whole inference costs a single forward pass per curve.","pith_inferences":["The same trained network might transfer to other break-prone extrapolation problems, such as learning curves in AutoML or hardware and financial scaling, where the segment structure applies.","Since the prior hyperparameters were manually fitted to the benchmark shapes, meta-learning or adapting them from the observed context would be a natural extension that could make the method robust to new curve families.","The cutoff restriction encodes a strong inductive assumption: after a downward segment, a future break is treated as unpredictable, and measuring whether predictive uncertainty rises on curves with late breaks would quantify what that assumption costs.","The interpolation objective opens a route to using the model as a multi-fidelity surrogate, deciding experimentally which scale to measure next rather than only extrapolating tails."],"forward_implications":["A single forward pass per curve replaces per-curve fitting, cutting inference from tens to hundreds of seconds (M4/BNSL/MCMC) to about 0.02 seconds while improving fit accuracy.","Small-scale observed context suffices to forecast the tail with a predictive distribution, so compute-budget decisions can weigh expected gain against cost and risk.","The model infers the number of breaks and the functional form implicitly, removing the cross-validation step that BNSL requires.","In Bayesian active learning, uncertainty-based selection of the next observation point improves predictive likelihood over MCMC baselines as more points are collected.","On double-descent curves, the method predicts the eventual downturn even when the context ends in an upward segment, which MCMC(BNSL) does not do."],"supporting_citations":[{"why":"Defines Prior-data Fitted Networks and the meta-learning objective that NSL-PFN adapts.","marker":"Müller et al., 2021"},{"why":"Introduces broken neural scaling laws, the break-based template and baseline that motivates the prior's random breaks.","marker":"Caballero et al., 2022"},{"why":"Supplies the M4 functional family and the image/language scaling-law benchmark datasets used for evaluation.","marker":"Alabdulmohsin et al., 2022"},{"why":"The LC-PFN baseline for learning-curve extrapolation that NSL-PFN compares against and improves upon.","marker":"Adriaensen et al., 2023"},{"why":"Provides the double-descent dataset and the empirical phenomenon that motivates upward segments in the prior.","marker":"Nakkiran et al., 2021"},{"why":"Supplies the MCMC sampler implementation that all MCMC baselines run on.","marker":"Foreman-Mackey et al., 2013"},{"why":"Introduces the M3 functional form used as one of the segment templates in the prior.","marker":"Bansal et al., 2022"},{"why":"Establishes the MCMC setup for learning-curve extrapolation that the Bayesian baselines follow.","marker":"Domhan et al., 2015"},{"why":"Source of the ColPret dataset of 192 scaling-law curves used as a large-scale evaluation set.","marker":"Choshen et al., 2024"}],"fun_headline_variants":["Meta-learned Bayesian model predicts scaling tails with uncertainty","PFN-based Bayesian extrapolation beats MCMC on scaling laws","Uncertainty-aware scaling extrapolation via prior-data fitted networks","Meta-learned prior yields calibrated uncertainty for scaling curves","Bayesian PFN extrapolates scaling laws with single forward pass"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation curves are all within the support of the manually designed prior, which was tuned by eye to match those same benchmarks, so the method's edge may depend on the prior already knowing the curve shapes.","fun_headline_variants_meta":{"raw":{"variants":["Meta-learned Bayesian model predicts scaling tails with uncertainty","PFN-based Bayesian extrapolation beats MCMC on scaling laws","Uncertainty-aware scaling extrapolation via prior-data fitted networks","Meta-learned prior yields calibrated uncertainty for scaling curves","Bayesian PFN extrapolates scaling laws with single forward pass"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000472,"raw_usage":{"total_tokens":2326,"prompt_tokens":907,"completion_tokens":1419,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":1336}},"tokens_in":523,"tokens_out":1419,"duration_ms":8587,"temperature":1.0,"reasoning_tokens":1336,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:55:13.417757+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate a family of scaling curves that is structurally absent from the Section 3.1 prior—for example, an exponential saturation curve with a long flat plateau at both ends, or a curve that keeps rising after a downward segment—and compare NSL-PFN against MCMC(BNSL) on RMSLE and MSCE; if NSL-PFN loses calibration or accuracy on these out-of-prior shapes, the prior-support assumption marks the limits of the claim.","supporting_citations":[{"cited_title":"Deep double descent: Where bigger models and more data hurt.Journal of Statistical Mechan- ics: Theory and Experiment, 2021(12):124003,","cited_arxiv_id":null,"evidence_quote":"Provides the double-descent dataset and the empirical phenomenon that motivates upward segments in the prior."}],"review_version":1}