{"id":"0b06c337-bf7f-424e-b867-d5017cf061d0","arxiv_id":"2505.23947","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"PFNs, which amortize Bayesian inference by training on datasets sampled from a prior, are likely to supersede MCMC and variational inference for most prediction tasks, the authors argue.","lead":"This position paper argues that Prior-data Fitted Networks (PFNs), already fast for small tabular data, will become the dominant approach to Bayesian prediction. A smart generalist should read it to understand a credible bet that synthetic pre-training can replace slow sampling or variational inference in low-data settings.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The dominance claim rests on treating PFN scaling and architectural failures as engineering issues, but the duplicate-counting failure in §6.5 is a representational limitation that undermines that premise.","rationale":"I read the paper as a position argument whose central claim is that PFNs will dominate most Bayesian prediction applications because amortized inference better exploits growing pre-training compute in data-scarce settings. For this claim to be more than an opinion, it needs PFN limitations to be engineering-level: larger contexts, faster inference, and better architectures should recover Bayesian behavior. The paper's own Section 3 flags this as a belief, and Section 6.5 gives a concrete failure that is not about compute. That is the most load-bearing soft spot. The reader identified the same general vulnerability (limitations treated as efficiency/compute-related rather than fundamental), so I partially agree. I differ by pointing to the duplicate-counting failure in §6.5 as the sharpest evidence, since it is already present in the paper and is a representational, not throughput, limitation. The Martingale plot in Figure 2 is also concerning but is admittedly small-scale; the counting failure is cleaner and more decisive. A single targeted experiment on duplicate counts would settle whether this is an isolated artifact or a fundamental barrier. I do not think the concern warrants changing the reader's conditional verdict: the paper is honest about its uncertainties and is a position paper, not a proof. The appropriate outcome remains a conditional acceptance with the explicit caveat that the dominance claim depends on resolving fundamental architectural limits, not just scaling compute.","tokens_in":19363,"tokens_out":3111,"duration_ms":38255,"concrete_test":"Re-run the §6.5 coin-flip experiment on TabPFN v2 and on the original PFN modified with zero-attention or positional encodings: condition on n identical heads for n = 1..100 and record the predicted head probability at each n. Then, on a real tabular benchmark with duplicated feature rows, compare PFN predictive updates against a conjugate exact posterior or full GP posterior. If the model's predictive distribution does not track the posterior as the duplicate count grows, the §6.5 failure is not an isolated architectural artifact, and the assumption that scaling limitations are purely efficiency-related should be rejected or the dominance claim should be substantially weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the position \"PFNs will dominate most applications of Bayesian prediction\" to hold, current PFN limitations must be addressable by more compute and better engineering. Section 3, Limitation 3, states this explicitly: \"While there are potentially fundamental limitations in learning from large-scale data in context, we believe most of the current reasons are efficiency- and compute-related.\" That belief is load-bearing. Section 6.5 and Figure 3 provide a concrete counterexample: an encoder-only transformer without positional embeddings cannot count identical examples. On a coin-flipping prior, conditioning on n identical heads should drive the posterior predictive probability toward 1, but the PFN prediction remains static. Counting duplicates is not a speed or compute problem; it is a failure of the architecture to represent a basic sufficient statistic required for Bayesian updating. Duplicate feature vectors occur in real tabular data, so this is not a pathological toy. The proposed fixes (zero attention, inference-time noise features) are speculative and are not shown to restore exact Bayesian behavior. The Martingale result in Figure 2 also shows a systematic early-rollout variance decrease that the authors themselves call \"not ... the case for a Bayesian predictor\"; the stabilization after a few steps is only demonstrated at very small scale. Thus the central claim depends on resolving a potentially fundamental behavioral limitation, not merely on faster inference or larger models.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This position paper argues that Prior-Data Fitted Networks (PFNs), trained on synthetic datasets sampled from a prior, will dominate most applications of Bayesian prediction and create new applications. It introduces the PFN formalism, reviews the prior-fitting objective, gives examples of priors, compares PFNs with MCMC/VI/GPs and amortized methods, discusses extensions (extra inputs, in-context interpreters, RL for BO, latent prediction), presents two small experiments (martingale property and duplicate counting), and proposes research directions for scaling, fast inference, and architectural improvements. It closes with alternative views and responses.","tokens_in":19615,"tokens_out":6148,"duration_ms":59475,"significance":"If the position is correct, Bayesian prediction would become a one-forward-pass operation, amortizing compute across datasets and enabling priors that are specified only through a sampler. The paper is strongest as a roadmap: it names concrete open problems (counting, heterogeneous features, interpretability, scaling) and points to published successes such as TabPFN v2, which has independent peer review. The paper is appropriately cautious in some places, but the strength of the dominance claim is not matched by the empirical evidence: the two new experiments are small-scale, and one of them (duplicate counting) exposes a representational limitation rather than an efficiency issue. The position is plausible and worth publishing, but the argument needs to be tightened before the strong claims can be endorsed.","major_comments":[{"comment":"The duplicate-counting experiment is a load-bearing counterexample to the claim in Section 3 (limitation 3) that current PFN shortcomings are mostly efficiency- and compute-related. On the coin-flip prior, the encoder-only transformer without positional embeddings cannot represent the count of identical observations, which is the sufficient statistic for the posterior; Figure 3 shows the predicted head probability remaining flat as the number of observed heads increases. Counting duplicated examples is a representational requirement, not a speed or compute issue, and duplicate feature vectors occur in real tabular data. The proposed remedies (zero attention, inference-time noise features) are mentioned only as suggestions and are not demonstrated to restore the correct posterior. The paper should either provide evidence that these architectural fixes recover Bayesian behavior on this prior, or substantially weaken the 'will dominate' claim.","section":"§6.5, Figure 3"},{"comment":"The martingale experiment shows a systematic decrease in predictive standard deviation over the first few rollout steps, which the authors themselves state 'should not be the case for a Bayesian predictor.' Since the central premise of the paper is that PFNs approximate the posterior predictive distribution, this deviation is directly relevant; calling it small-scale does not remove the need to explain it. The paper should report a more systematic evaluation (multiple priors, longer rollouts, statistical significance) and either reconcile the deviation with the Bayesian claim or qualify the scope of the claim in Section 1 and the Position box.","section":"§5.1, Figure 2"},{"comment":"The claim that PFNs will dominate most Bayesian prediction applications assumes that scaling limitations are engineering challenges. Section 6.1 offers architectural ideas (Perceiver, linear attention, state-space models) but no results showing that these preserve the prior-fitting guarantee at larger sample sizes. Since Section 2.1's approximation guarantee is stated for a fixed architecture trained on a fixed prior, it is not automatic that changing the architecture preserves posterior predictive approximation; this transfer should be demonstrated or explicitly discussed as an open assumption. Without this, the response to View 2 remains speculative, so the dominance claim is not fully supported.","section":"§6.1, §7 View 2"}],"minor_comments":[{"comment":"The sentence on the BNN prior contains a grammatical error ('it's weight') and the notation 'N(0, σ2)' is not tied to the σ introduced in the text.","section":"§2.2"},{"comment":"'knowledege' is a typo; it should read 'knowledge'.","section":"§6.1"},{"comment":"'add zero attnin PyTorch' has a typo; it should read 'add zero attention in PyTorch'.","section":"§6.5"},{"comment":"The caption describes training on 'sines and sloped lines' and 'slightly sloped sines', but the surrounding text and the rest of the appendix concern Martingale rollouts; the caption appears to be from a different experiment and should be corrected.","section":"Appendix B, Figure 4 caption"},{"comment":"The bullet 'No Possibility for Data Leakage' is too absolute; leakage can occur through the choice of prior, preprocessing, or evaluation protocol, even if pre-training data itself is synthetic.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is essentially an extended position statement by the inventors of TabPFN, with the empirical backbone drawn largely from their own prior publications. This is not disqualifying because TabPFN v2 was independently peer-reviewed in Nature and the paper is transparent about its advocacy; still, for a journal-level endorsement the editor may wish to see a more balanced treatment of competing amortized and non-amortized approaches and possibly an independent demonstration of the counting and martingale issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you want the most mature statement of the PFN research program to date. The paper is not a technical breakthrough; it is a position piece with two small new experiments. And it works well as a position piece: the authors state their case plainly, list limitations without much spin, and propose concrete research directions. The counting experiment (duplicate inputs, static predictions) is a genuine negative result, and the Martingale test applied to PFNs is a useful extension of earlier LLM-focused work.\n\nThe soft spot is the load-bearing assumption behind “PFNs will dominate most applications of Bayesian prediction.” Section 3, limitation 3, attributes scale issues to “efficiency- and compute-related” reasons. But the counting failure in Section 6.5 is not a speed or compute problem; it is a representational limitation. An encoder-only transformer without positional embeddings cannot count identical examples, which means it cannot update on a basic sufficient statistic. The proposed fixes (zero attention, inference-time noise) are speculative and untested. That does not sink the position paper, but it does weaken the inference that current limitations are engineering hurdles rather than potentially deeper architectural gaps. The Martingale result similarly shows a systematic early-rollout deviation from Bayesian behavior; the authors acknowledge it but lean on the stabilization at small scale.\n\nCredit where due: the paper does not hide its weaknesses. The limitations section is unusually honest, and the authors are transparent that the empirical demonstrations are small-scale. The core argument is clearly an opinion, not a proof, and the literature survey is broad enough to give the reader context.\n\nWho is this for? Researchers already working with PFNs or amortized inference, and anyone considering whether to build on this line of work. It will generate discussion, and it deserves a serious referee. I would send it to peer review with the caveat that the reviewers should push for a more balanced interpretation of the Martingale data and a more careful separation of current engineering issues from representational limitations that might be fundamental. The authors do not need to prove dominance; they do need to stop implying the evidence is stronger than it is.","headline":"A clear, honest position paper from the PFN inventors, but the dominance claim rests on an assumption about limitations that the paper's own counting experiment undercuts.","tokens_in":20179,"tokens_out":2338,"would_cite":false,"duration_ms":27672,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","62F15"],"pacs":[],"model":"deepseek-v4-flash","headline":"Bayesian prediction may become a neural forward pass: prior-fitted networks trained on synthetic data replace per-dataset MCMC and variational inference with a single amortized forward pass.","keywords":["Prior-Data Fitted Networks","Bayesian prediction","amortized inference","in-context learning","synthetic data","posterior predictive distribution","tabular data","Bayesian optimization"],"falsifier":"Train a PFN on a prior with a known posterior predictive (e.g., a Gaussian process or linear regression with a conjugate prior), then measure the KL divergence between the PFN's predictive distribution and the exact posterior predictive as the in-context dataset size grows within the prior's support; if increasing model scale and pre-training compute by fixed factors leaves a non-shrinking KL gap, or if the martingale violation observed in the paper's Figure 2 grows with scale instead of stabilizing, the dominance claim is falsified. A sharper version: find any prior from which the PFN can sample such that no scaling of the current architecture class achieves near-Bayesian predictions on all datasets in the prior's support.","tokens_in":19147,"feed_emoji":"🧠","tokens_out":4154,"duration_ms":38144,"temperature":0.7,"pith_summary":"This position paper argues that Bayesian prediction will increasingly be performed by Prior-Data Fitted Networks (PFNs): neural networks pre-trained on millions of datasets sampled from a user-defined prior, which then approximate the posterior predictive distribution in a single forward pass. The rationale is that pre-training compute is scaling faster than the supply of real-world data, so the efficient division of labor is to spend compute once on a prior and amortize it across every future dataset drawn from that prior. If the authors are right, the standard toolkit of Bayesian prediction—MCMC sampling, variational inference, and Gaussian-process marginalization—would be superseded for most supervised-prediction tasks by a standardized neural-network forward pass. The paper also maps the open problems—interpretability, scaling to large datasets, inference speed, architectural counting limits—and argues that most are engineering obstacles rather than fundamental barriers.","feed_headline":"Bayesian prediction may become a neural forward pass","feed_subtitle":"Pre-training on synthetic data lets one network answer any prediction from that prior in a single pass, no MCMC or VI.","key_machinery":"The load-bearing object is the prior-fitting objective: sampling datasets $D \\sim p(D)$ from a prior over datasets and optimizing the network parameters $\\theta$ to minimize $\\ell = \\mathbb{E}_{D \\sim p(D)}[-\\log q_\\theta(y_{\\text{test}} | x_{\\text{test}}, D_{\\text{train}})]$, which equals the expected KL divergence between the true posterior predictive $p(\\cdot | x, D)$ and the network approximation $q_\\theta(\\cdot | x, D)$ up to a constant. The prior is supplied as a sampling program—for example, weights of a Bayesian neural network, Gaussian-process hyperparameters, or a structural causal model as in TabPFN—which means the user declares the inductive bias without needing tractable densities. The forward pass with masked attention, where training points attend to each other and the test point attends only to training points, then performs amortized Bayesian inference by in-context learning.","core_discovery":"On the authors' own terms, the central claim is that PFNs will dominate most applications of Bayesian prediction and create new ones. A PFN is trained by minimizing the cross-entropy of held-out examples across datasets sampled from a prior, which is exactly the KL divergence from the network's predictive distribution to the true posterior predictive distribution; the prior itself is defined declaratively by a synthetic data-generating process rather than by a tractable likelihood. Because the network is conditioned only on the observed dataset and query at inference time, prediction becomes in-context learning: no latent posterior is instantiated, no per-dataset sampling is run, and the same amortized model serves any dataset from the prior. The authors point to TabPFN as the poster child, reporting that it outperforms tuned classical baselines on small tabular datasets, and to applications in Bayesian optimization, learning-curve extrapolation, time series, and biology as evidence that the amortization pattern transfers across domains.","pith_inferences":["If the amortization thesis holds, the economic structure of Bayesian practice flips: the scarce resource becomes prior engineering (writing good sampling programs) rather than inference engineering (converging samplers or tuning variational families).","A testable extension is that PFN performance on a given prior should track a scaling law in pre-training compute and training datasets, with deviations from the true posterior predictive shrinking monotonically—something the paper does not yet demonstrate.","The counting failure on identical examples (Section 6.5) suggests a boundary condition: architectures without positional or counting mechanisms cannot represent sufficient statistics that depend on multiplicities, so the dominance claim implicitly requires architectural progress of exactly the kind the paper lists as open."],"forward_implications":["Bayesian prediction in data-scarce domains becomes a forward pass on a pre-trained network, removing the per-dataset cost of MCMC or variational inference.","Priors that are only defined by a sampling process—complex graphs, architectures, simulators—become usable for Bayesian prediction, since no density evaluation is needed.","The same pre-trained PFN can serve many datasets from one prior, shifting compute from inference time to pre-training time, which suits the trend of falling pre-training cost and stagnant real-world data growth.","Application areas such as tabular learning, Bayesian optimization, learning-curve extrapolation, and time-series forecasting already exhibit this pattern, and more domains should follow as the efficiency gaps are closed."],"supporting_citations":[{"why":"Provides the original PFN formulation and the prior-fitting objective that the whole argument builds on.","marker":"(Müller et al., 2022)"},{"why":"TabPFN, the paper's main empirical success story, first consistently outperforming XGBoost on small tabular datasets.","marker":"(Hollmann et al., 2023)"},{"why":"TabPFN v2 scaling to 10,000 examples with fast caching; the strongest evidence for the scalability and speed claims.","marker":"(Hollmann et al., 2025)"},{"why":"Shows over 10,000x speedup for learning-curve extrapolation, supporting the amortization-across-domains claim.","marker":"(Adriaensen et al., 2023)"},{"why":"Introduces the martingale property used in Section 5.1 to test whether in-context learners behave as Bayesian predictors.","marker":"(Falck et al., 2024)"},{"why":"Provides statistical foundations and initial analysis of PFN asymptotics, cited for the limit-behavior discussion.","marker":"(Nagler, 2023)"},{"why":"Proves in-context learning behavior for transformers, underpinning the claim that PFNs can perform Bayesian prediction.","marker":"(Bai et al., 2024)"},{"why":"PFNs4BO, demonstrating PFNs applied to Bayesian optimization and supporting the domain-expansion argument.","marker":"(Müller et al., 2023c)"}],"fun_headline_variants":["Prior-fitted networks: Bayesian prediction in a single pass","Synthetic pre-training enables Bayesian prediction in one forward pass","Amortized Bayesian inference: pre-train once, predict any dataset","One neural network for any Bayesian query, no per-dataset sampling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim stands on the assumption that the current PFN weaknesses—small-support effectiveness only, slow inference, and failures like not counting repeated identical examples—are efficiency and engineering problems that scale away with compute and architectural tweaks, rather than fundamental limits of amortized in-context prediction.","fun_headline_variants_meta":{"raw":{"variants":["Prior-fitted networks: Bayesian prediction in a single pass","Synthetic pre-training enables Bayesian prediction in one forward pass","Amortized Bayesian inference: pre-train once, predict any dataset","One neural network for any Bayesian query, no per-dataset sampling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000923,"raw_usage":{"total_tokens":3931,"prompt_tokens":891,"completion_tokens":3040,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":2968}},"tokens_in":507,"tokens_out":3040,"duration_ms":21701,"temperature":1.0,"reasoning_tokens":2968,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:37:10.575442+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a PFN on a prior with a known posterior predictive (e.g., a Gaussian process or linear regression with a conjugate prior), then measure the KL divergence between the PFN's predictive distribution and the exact posterior predictive as the in-context dataset size grows within the prior's support; if increasing model scale and pre-training compute by fixed factors leaves a non-shrinking KL gap, or if the martingale violation observed in the paper's Figure 2 grows with scale instead of stabilizing, the dominance claim is falsified. A sharper version: find any prior from which the PFN can sample such that no scaling of the current architecture class achieves near-Bayesian predictions on all datasets in the prior's support.","supporting_citations":[{"cited_title":"Tab PFN : A transformer that solves small tabular classification problems in a second","cited_arxiv_id":null,"evidence_quote":"TabPFN, the paper's main empirical success story, first consistently outperforming XGBoost on small tabular datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the martingale property used in Section 5.1 to test whether in-context learners behave as Bayesian predictors."}],"review_version":1}