REVIEW 4 major objections 5 minor 1 cited by
On Finetuning Tabular Foundation Models
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Finetuning improves the tabular foundation model TabPFNv2 by refining the query–key similarity scores in its last-layer attention, so that the model attends to the training rows whose labels actually matter.
desk verdict A useful, honest finetuning study for TabPFNv2 whose practical recipe holds up, but the mechanism claim is correlational and the abstract overstates it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the last-layer inter-sample attention of TabPFNv2, a tabular foundation model that predicts each test row by attending over the training rows provided in its input context. The query-key dot products in this attention layer act as a learned similarity measure between a test object and each in-context training object, and the attention softmax weights the corresponding training labels to form the prediction. The paper reads this mechanism as an implicit retrieval system, analogous to explicitly retrieval-based models, and validates that reading by using the attention weights alone as a weighted-$k$NN predictor: if the weights truly reflect target closeness, that predictor should be accurate. Finetuning is shown to improve the model precisely by sharpening these dot products, so the same attention machinery produces a better-matching set of neighbors.
What would settle it
Take a finetuned TabPFNv2 and freeze everything except randomly perturbing the last-layer attention projections; if test accuracy barely changes while the attention weights are destroyed, the refined query-key dot products are not the causal channel. A cleaner check: replace the trained attention scores on test samples with the finetuned dot products computed from a model whose last layer has been reinitialized; if this recovers most of the finetuning improvement, the similarity signal carries the gain, and if it does not, the gain lives elsewhere.
Extended reading notes
Core claim
The central claim is that the benefit of finetuning TabPFNv2 can be traced to a single mechanism: the model's last-layer inter-sample attention, which is used to retrieve relevant training examples and weight their labels, produces query-key dot products that are more faithful to the true target similarity after finetuning. The paper supports this with a direct test: using only the attention weights as a weighted-nearest-neighbor predictor over training labels, the finetuned weights approximate test targets far better than the pretrained weights, and in most datasets the largest prediction improvements occur exactly on the test samples whose attention distribution becomes more concentrated after finetuning. In the authors' view, finetuning does not teach the model new features or a better output head; it repairs the similarity metric that the existing in-context retrieval logic relies on.
Load-bearing premise
The diagnosis assumes that TabPFNv2's predictions can be read off from its last-layer attention weights, so that changes in those weights explain the performance gains; the paper supports this with correlations and weight-based proxy predictions, not by intervening on the attention mechanism directly.
Editorial extensions
If this is right
- Full finetuning, with the right learning-rate schedule, should be the default adaptation method for TabPFNv2 on datasets up to tens of thousands of rows; LoRA and partial-update schemes add complexity without clear accuracy gains.
- Because the gains come from the similarity signal, future attempts to improve TabPFNv2 adaptation can focus on the metric that drives the last-layer attention rather than on new feature embeddings or prediction heads.
- On I.I.D. academic benchmarks, finetuned TabPFNv2, especially in ensembles, becomes competitive with or better than current non-foundational tabular deep-learning models.
- On datasets with gradual temporal shift and rich feature sets, in-context and finetuned TabPFNv2 are less stable, and non-foundational methods like TabM retain an advantage.
Reading between the lines
- The paper's own evidence is correlational, so an intervention is the natural next test: if the query-key dot products are the causal channel, artificially sharpening or reweighting them at inference time should reproduce part of the finetuning gain without any gradient updates.
- The finding that finetuning helps more on larger datasets suggests the pretrained similarity metric degrades relative to what the data can support; this predicts that the gap between finetuned and in-context TabPFNv2 should widen further as dataset size grows, up to the model's context limit.
- The attention-entropy diagnostic could be turned into a practical tool: monitoring the shift in last-layer attention entropy on a validation set during finetuning may give an early signal of whether adaptation is actually improving the retrieval mechanism.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies gradient-based adaptation of TabPFNv2, a tabular foundation model. In the first part, it systematically compares full finetuning, parameter-efficient methods (LoRA, last layers, embeddings/LayerNorm/head), and no-finetuning on a benchmark of medium-scale academic datasets, concluding that full finetuning is the most time-efficient and overall strongest practical baseline. The second part proposes a mechanistic explanation: finetuning improves the last-layer query-key dot products between test and training objects so that they better reflect target similarity, thereby sharpening an implicit retrieval mechanism. The paper then compares TabPFNv2 and its finetuned variants against modern tabular DL baselines on academic datasets and on subsampled TabReD datasets, finding that finetuned TabPFNv2 reaches state-of-the-art results on the academic benchmark while being less stable on temporally shifted, feature-rich TabReD data.
Significance. If the empirical findings hold, the practical contribution is valuable: a reproducible, well-tuned full-finetuning recipe for TabPFNv2, a stronger benchmark comparison than in the original TabPFNv2 paper, and a clear statement of scalability limits. The study uses multiple seeds, systematic hyperparameter tuning, a public code release, and an unusually honest limitations section, which strengthens confidence in the empirical part. The mechanistic claim, however, is currently supported only by correlational diagnostics and by a proxy that bypasses the model's actual output path; this part would need either an intervention or substantially weaker causal language to be considered established. As stated, the abstract's 'stems from' overstates the evidence, even though the practical recommendation (full finetuning works well) is independent of the mechanism and appears sound.
major comments (4)
- [§4, Table 3] The central mechanistic claim is supported only by a correlational, partly by-construction diagnostic. Table 3 measures attention quality by using last-layer attention weights to average raw training targets (or class logits), which is not TabPFNv2's actual prediction path: the model uses attention to combine value representations before passing them through a prediction head. Moreover, the attention weights come from a model finetuned on the same training targets, so improved alignment between query-key dot products and target similarity is expected whenever any representation shift improves prediction; it does not establish that refined dot products cause the performance gain. To support the causal claim in the abstract and in Contribution 2, the paper needs an intervention that changes attention weights independently of other components (for example, freezing or perturbing query/key projections while keeping values and the head fixed) or a mediation analysis that separates the contribution of attention weights from other learned components.
- [§4, Figure 4] The sample-wise relationship between entropy change and error change is correlational, and all model parameters are updated jointly, so the observed co-occurrence of sharper attention and improved error is consistent with both the proposed mechanism and any alternative explanation in which task-aligned representations improve predictions while attention sharpening is epiphenomenal. The claim that 'the most performance gains are obtained from those samples where entropy dropped' is not quantified: Figure 4 shows a smoothed trend, but there is no aggregate statistic reporting the share of total error reduction attributable to entropy-decreasing samples, nor a statistical test of the relationship. The Churn dataset is an acknowledged exception, but its existence weakens the generality of the proposed mechanism and deserves a more substantive discussion than a single sentence pointing to future work.
- [Abstract and §4] The abstract and Contribution 2 state that finetuning's success 'stems from' improved query-key dot products and that the paper 'reveals' this, whereas Section 4 itself introduces the retrieval mechanism as a 'conjecture' and the finetuning effect as a 'hypothesize[d]' refinement of similarity signals. This is not merely a wording issue: the causal claim is a stated contribution, and the evidence presented is correlational. The language should be aligned with the evidence, either by replacing 'stems from' with formulations such as 'is consistent with' or 'is accompanied by,' or by adding the missing intervention/mediation evidence that would justify the stronger claim.
- [§6, Limitations] The limitations section is candid about dataset-scale and preprocessing choices but omits the main epistemic limitation of the paper: the mechanistic conclusion in §4 is based on correlational diagnostics and a proxy prediction scheme, not on evidence about the causal role of attention weights in the model's own predictions. Since the manuscript explicitly enumerates limitations, this missing caveat should be added; it is load-bearing for the paper's second contribution and for the abstract's causal claim.
minor comments (5)
- [Table 3] Table 3 reports scores from a single seed; given that the rest of the paper uses multiple seeds, the attention-proxy comparison should include variance or a multi-seed average to rule out seed-dependent conclusions.
- [§4, Figure 4] The x-axis construction is described only in Appendix B and is initially confusing: the figure says 'Index of test sample, sorted by ΔH,' but the red line marking ΔH=0 is not explained in the main text. Please add a one-sentence explanation in the caption or main text.
- [Appendix B] There are several typos in the appendix and figure captions: 'retreival-based' (§4), 'concetrated' and 'entopy' (Appendix B), 'explaination' (Figure 3 caption), and 'classifiation' (Table 4). These should be corrected.
- [References] The reference to Von Oswald et al. is rendered as 'V on Oswald' and should be fixed.
- [§3, Table 2] The table title says 'Pred. Length' but the text describes it as batch size; please clarify in the caption that the prediction sequence length is the number of objects used to compute the loss per gradient step.
Circularity Check
No forced circularity: the finetuning comparison is an independent empirical result, while the mechanistic claim is openly labeled a conjecture supported by correlational diagnostics rather than an equation-level reduction.
full rationale
The paper's central empirical claim—that full finetuning of TabPFNv2 is practically effective and often achieves state-of-the-art results on medium-scale academic datasets—is an independent finding based on held-out test evaluations, with no fitted parameter renamed as a prediction. The mechanistic proposal in Section 4 is explicitly introduced as a conjecture ('we conjecture that an important part of TabPFNv2 prediction mechanics is an implicit retrieval mechanism'), and the supporting evidence (Table 3, Figures 3 and 4) is correlational: attention weights after finetuning produce better weighted-kNN proxies and lower entropy on many samples, but the paper does not claim this diagnostic is an intervention proving causation; the abstract's 'stems from' is an interpretation, not a derivation. The retrieval analogy is motivated by a correlation with ModernNCA (0.89 Pearson), which is an external baseline rather than a self-citation, and the paper does not rely on the authors' own prior theorems or on an ansatz smuggled through self-citation. One limitation is honestly stated (Churn does not align with the entropy story), and the paper notes the lack of causal proof by describing the mechanism as conjectural. No equation-level reduction, no fitted-input-as-prediction, and no load-bearing self-citation chain were found. The score of 2 reflects the minor epistemic gap between correlational diagnostics and the causal-sounding 'stems from' wording, not a structural circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption TabPFNv2 pretrained checkpoints and architecture (Hollmann et al., 2025) are valid and used as-is.
- ad hoc to paper Last-layer inter-sample attention scores are a sufficient proxy for TabPFNv2's retrieval-based prediction mechanism.
- domain assumption The benchmark datasets (Grinsztajn et al., 2022; Gorishniy et al., 2021; TabReD subsamples) are representative of small-to-medium tabular problems, and each baseline's preprocessing is near-optimal.
- domain assumption The synthetic pretraining distribution used for TabPFNv2 provides a useful prior for these downstream datasets.
Cite this review
Pith. "Pith review of On Finetuning Tabular Foundation Models." pith.science (2026). https://pith.science/paper/L22V7M2H
@misc{pith2026250608982,
author = {Pith},
title = {Pith review of: On Finetuning Tabular Foundation Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/L22V7M2H}},
note = {Machine review of arXiv:2506.08982}
}
read the original abstract
Foundation models are an emerging research direction in tabular deep learning. Notably, TabPFNv2 recently claimed superior performance over traditional GBDT-based methods on small-scale datasets using an in-context learning paradigm, which does not adapt model parameters to target datasets. However, the optimal finetuning approach for adapting tabular foundational models, and how this adaptation reshapes their internal mechanisms, remains underexplored. While prior works studied finetuning for earlier foundational models, inconsistent findings and TabPFNv2's unique architecture necessitate fresh investigation. To address these questions, we first systematically evaluate various finetuning strategies on diverse datasets. Our findings establish full finetuning as the most practical solution for TabPFNv2 in terms of time-efficiency and effectiveness. We then investigate how finetuning alters TabPFNv2's inner mechanisms, drawing an analogy to retrieval-augmented models. We reveal that the success of finetuning stems from the fact that after gradient-based adaptation, the dot products of the query-representations of test objects and the key-representations of in-context training objects more accurately reflect their target similarity. This improved similarity allows finetuned TabPFNv2 to better approximate target dependency by appropriately weighting relevant in-context samples, improving the retrieval-based prediction logic. From the practical perspective, we managed to finetune TabPFNv2 on datasets with up to 50K objects, observing performance improvements on almost all tasks. More precisely, on academic datasets with I.I.D. splits, finetuning allows TabPFNv2 to achieve state-of-the-art results, while on datasets with gradual temporal shifts and rich feature sets, TabPFNv2 is less stable and prior methods remain better.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 1 Pith paper
-
Turning Tabular Foundation Models into Graph Foundation Models
G2T-FM converts graph node tasks into tabular tasks and shows that tabular foundation models can match or beat well-tuned GNNs, especially after finetuning.
Reference graph
Works this paper leans on
-
[3]
den Breejen, F., Bae, S., Cha, S., Kim, T.-Y ., Koh, S
URLhttps://openreview.net/forum?id=6PmJoRfdaK. den Breejen, F., Bae, S., Cha, S., Kim, T.-Y ., Koh, S. H., and Yun, S.-Y . Fine-tuning the retrieval mechanism for tabular deep learning. InNeurIPS 2023 Second Table Representation Learning Workshop,
work page 2023
-
[5]
Howard, J. and Ruder, S. Universal language model fine-tuning for text classification. InACL 2018-56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers), volume 1, pp. 328–339. Association for Computational Linguistics,
work page 2018
-
[10]
Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. A. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.Advances in Neural Information Processing Systems, 35:1950–1965,
work page 1950
-
[12]
Prokhorenkova, L., Gusev, G., V orobev, A., Dorogush, A
https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html. Prokhorenkova, L., Gusev, G., V orobev, A., Dorogush, A. V ., and Gulin, A. CatBoost: unbiased boosting with categorical features. InNeurIPS,
work page 2022
-
[13]
Revisiting pretraining objectives for tabular deep learning.arXiv, 2207.03208v1,
Rubachev, I., Alekberov, A., Gorishniy, Y ., and Babenko, A. Revisiting pretraining objectives for tabular deep learning.arXiv, 2207.03208v1,
-
[15]
Retrieval & fine-tuning for in-context tabular models.arXiv preprint arXiv:2406.05207,
Thomas, V ., Ma, J., Hosseinzadeh, R., Golestan, K., Yu, G., V olkovs, M., and Caterini, A. Retrieval & fine-tuning for in-context tabular models.arXiv preprint arXiv:2406.05207,
-
[16]
Mixture of in-context prompters for tabular pfns
11 Xu, D., Cirit, O., Asadi, R., Sun, Y ., and Wang, W. Mixture of in-context prompters for tabular pfns. arXiv preprint arXiv:2405.16156,
-
[17]
Ye, H.-J., Yin, H.-H., and Zhan, D.-C. Modern neighborhood components analysis: A deep tabular baseline two decades later.arXiv, 2407.03257v1,
Show all 20 references
-
[18]
T., Wang, F., Yan, Y ., Shen, X., and Zhang, Q
Yin, Q., He, X., Deng, L., Leong, C. T., Wang, F., Yan, Y ., Shen, X., and Zhang, Q. Deeper insights without updates: The power of in-context learning over fine-tuning.arXiv preprint arXiv:2410.04691,
-
[19]
To run it one must also obtain the TabPFNv2 checkpoints from hf.co/Prior-Labs
A Reproducibility Statement We provide the code for TabPFNv2 finetuning on github.com:yandex-research/tabpfn-finetuning. To run it one must also obtain the TabPFNv2 checkpoints from hf.co/Prior-Labs. We reuse the datasets from (Gorishniy et al., 2024). See the bin/tabpfnv2_fin...
2024
-
[20]
A more thorough analysis of such cases is reserved for future work
This observation motivates deeper investigation into the inner workings of the model, and the particular mechanisms which improve the latent space for calculating similarity between objects. A more thorough analysis of such cases is reserved for future work. C Datasets and Ext...
2022
-
[2017]
Big transfer (bit): General visual representation learning
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learning. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, pp. 491–507. Springer,
2020
-
[2018]
J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W
10 Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685,
-
[2019]
The power of scale for parameter-efficient prompt tuning
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691,
-
[2020]
What would elsa do? freezing layers during transformer fine-tuning
Lee, J., Tang, R., and Lin, J. What would elsa do? freezing layers during transformer fine-tuning. arXiv preprint arXiv:1911.03090,
1911 arXiv
-
[2021]
S., Kolesnikov, A., Wang, X., Salz, D., Neumann, M., Alabdulmohsin, I., Tschannen, M., Bugliarello, E., et al
Beyer, L., Steiner, A., Pinto, A. S., Kolesnikov, A., Wang, X., Salz, D., Neumann, M., Alabdulmohsin, I., Tschannen, M., Bugliarello, E., et al. Paligemma: A versatile 3b vlm for transfer.arXiv preprint arXiv:2407.07726,
-
[2022]
In-context data distillation with tabpfn.arXiv preprint arXiv:2402.06971,
Ma, J., Thomas, V ., Yu, G., and Caterini, A. In-context data distillation with tabpfn.arXiv preprint arXiv:2402.06971,
-
[2023]
T., Cherepanova, V ., Hegde, C., Hutter, F., Goldblum, M., Cohen, N., and White, C
Feuer, B., Schirrmeister, R. T., Cherepanova, V ., Hegde, C., Hutter, F., Goldblum, M., Cohen, N., and White, C. Tunetables: Context optimization for scalable prior-data fitted networks.arXiv preprint arXiv:2402.11137,
-
[2024]
D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,
1901
-
[2025]
B., and Goldstein, T
Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C. B., and Goldstein, T. SAINT: improved neural networks for tabular data via row attention and contrastive pre-training.arXiv, 2106.01342v1,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.