{"id":"767404db-a07a-4b7c-b016-7c1856d67637","arxiv_id":"2506.08982","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Full finetuning of TabPFNv2 outperforms in-context learning and partial finetuning on medium tabular datasets, and its gains come from sharper query-key attention that better reflects target similarity.","lead":"The paper evaluates different ways to adapt the TabPFNv2 tabular foundation model to new datasets and finds that full finetuning, updating all parameters, works best on medium-sized tables. It argues that finetuning improves the model's internal similarity scoring, which makes its attention-based retrieval more accurate.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central mechanistic claim is supported only by correlational diagnostics; no intervention shows that refined last-layer attention weights cause the finetuning gains, so the abstract's causal 'stems from' overstates the evidence.","rationale":"The reader identifies the same load-bearing weakness: the mechanistic conclusion rests on correlational evidence, and no intervention manipulates attention independently to establish causation. I agree, and the concern is genuinely load-bearing because the abstract frames the mechanism causally ('stems from'), while the evidence in §4 is an attention-weighted kNN proxy plus entropy/error correlations. The practical recommendation—full finetuning as a strong baseline—is well supported by the systematic comparison and is not undermined by this issue. The reader's CONDITIONAL verdict already captures the appropriate stance: the paper should either soften the causal language or add an attention-patch intervention. No novel concern beyond the reader's was found, so no verdict adjustment is needed.","tokens_in":23518,"tokens_out":7395,"duration_ms":81439,"concrete_test":"Perform a causal attention-patch experiment on at least four benchmark datasets (e.g., California, House, Adult, Diamond) using the released code: take the fully finetuned model and, at inference, replace only the last-layer inter-sample attention probability matrix with the corresponding matrix from the original pretrained model, leaving all other finetuned parameters and activations unchanged. If this patched model's performance drops back to approximately the No-FT level, the refined attention weights are causally necessary for the gains; if it retains most of the full-finetuning improvement, the central claim fails because other components mediate the gains. Run the reverse patch (pretrained model with finetuned attention) as a complement. This directly tests whether improved query-key similarity is the mechanism or merely a correlate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Abstract and §4 claim finetuning's success 'stems from' improved query-key dot products in the last layer. The evidence is correlational: Table 3 shows attention-weighted kNN predictions improve after finetuning; Figure 3 shows attention entropy drops; Figure 4 shows larger error improvements where entropy drops. None of these distinguishes the proposed mechanism from an epiphenomenon. Finetuning updates all parameters, so any task-aligned representation shift will also make attention weights more target-aligned, regardless of whether the performance gain is actually mediated by value vectors, the MLP head, or earlier layers. Moreover, the Table 3 diagnostic bypasses the model's real output path: it uses attention weights to average raw training targets, while TabPFNv2 uses attention to combine value representations before a prediction head. An improved proxy for target similarity does not demonstrate that this similarity is the causal driver of the model's own predictions. Section 4 explicitly labels the retrieval mechanism a 'conjecture.' A causal claim requires an intervention that changes attention weights independently while holding other components fixed. Without such evidence, the strongest abstract claim is unverified, even though the empirical finetuning comparison and practical recommendation appear solid.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies gradient-based adaptation of TabPFNv2, a tabular foundation model. In the first part, it systematically compares full finetuning, parameter-efficient methods (LoRA, last layers, embeddings/LayerNorm/head), and no-finetuning on a benchmark of medium-scale academic datasets, concluding that full finetuning is the most time-efficient and overall strongest practical baseline. The second part proposes a mechanistic explanation: finetuning improves the last-layer query-key dot products between test and training objects so that they better reflect target similarity, thereby sharpening an implicit retrieval mechanism. The paper then compares TabPFNv2 and its finetuned variants against modern tabular DL baselines on academic datasets and on subsampled TabReD datasets, finding that finetuned TabPFNv2 reaches state-of-the-art results on the academic benchmark while being less stable on temporally shifted, feature-rich TabReD data.","tokens_in":23756,"tokens_out":4473,"duration_ms":49932,"significance":"If the empirical findings hold, the practical contribution is valuable: a reproducible, well-tuned full-finetuning recipe for TabPFNv2, a stronger benchmark comparison than in the original TabPFNv2 paper, and a clear statement of scalability limits. The study uses multiple seeds, systematic hyperparameter tuning, a public code release, and an unusually honest limitations section, which strengthens confidence in the empirical part. The mechanistic claim, however, is currently supported only by correlational diagnostics and by a proxy that bypasses the model's actual output path; this part would need either an intervention or substantially weaker causal language to be considered established. As stated, the abstract's 'stems from' overstates the evidence, even though the practical recommendation (full finetuning works well) is independent of the mechanism and appears sound.","major_comments":[{"comment":"The central mechanistic claim is supported only by a correlational, partly by-construction diagnostic. Table 3 measures attention quality by using last-layer attention weights to average raw training targets (or class logits), which is not TabPFNv2's actual prediction path: the model uses attention to combine value representations before passing them through a prediction head. Moreover, the attention weights come from a model finetuned on the same training targets, so improved alignment between query-key dot products and target similarity is expected whenever any representation shift improves prediction; it does not establish that refined dot products cause the performance gain. To support the causal claim in the abstract and in Contribution 2, the paper needs an intervention that changes attention weights independently of other components (for example, freezing or perturbing query/key projections while keeping values and the head fixed) or a mediation analysis that separates the contribution of attention weights from other learned components.","section":"§4, Table 3"},{"comment":"The sample-wise relationship between entropy change and error change is correlational, and all model parameters are updated jointly, so the observed co-occurrence of sharper attention and improved error is consistent with both the proposed mechanism and any alternative explanation in which task-aligned representations improve predictions while attention sharpening is epiphenomenal. The claim that 'the most performance gains are obtained from those samples where entropy dropped' is not quantified: Figure 4 shows a smoothed trend, but there is no aggregate statistic reporting the share of total error reduction attributable to entropy-decreasing samples, nor a statistical test of the relationship. The Churn dataset is an acknowledged exception, but its existence weakens the generality of the proposed mechanism and deserves a more substantive discussion than a single sentence pointing to future work.","section":"§4, Figure 4"},{"comment":"The abstract and Contribution 2 state that finetuning's success 'stems from' improved query-key dot products and that the paper 'reveals' this, whereas Section 4 itself introduces the retrieval mechanism as a 'conjecture' and the finetuning effect as a 'hypothesize[d]' refinement of similarity signals. This is not merely a wording issue: the causal claim is a stated contribution, and the evidence presented is correlational. The language should be aligned with the evidence, either by replacing 'stems from' with formulations such as 'is consistent with' or 'is accompanied by,' or by adding the missing intervention/mediation evidence that would justify the stronger claim.","section":"Abstract and §4"},{"comment":"The limitations section is candid about dataset-scale and preprocessing choices but omits the main epistemic limitation of the paper: the mechanistic conclusion in §4 is based on correlational diagnostics and a proxy prediction scheme, not on evidence about the causal role of attention weights in the model's own predictions. Since the manuscript explicitly enumerates limitations, this missing caveat should be added; it is load-bearing for the paper's second contribution and for the abstract's causal claim.","section":"§6, Limitations"}],"minor_comments":[{"comment":"Table 3 reports scores from a single seed; given that the rest of the paper uses multiple seeds, the attention-proxy comparison should include variance or a multi-seed average to rule out seed-dependent conclusions.","section":"Table 3"},{"comment":"The x-axis construction is described only in Appendix B and is initially confusing: the figure says 'Index of test sample, sorted by ΔH,' but the red line marking ΔH=0 is not explained in the main text. Please add a one-sentence explanation in the caption or main text.","section":"§4, Figure 4"},{"comment":"There are several typos in the appendix and figure captions: 'retreival-based' (§4), 'concetrated' and 'entopy' (Appendix B), 'explaination' (Figure 3 caption), and 'classifiation' (Table 4). These should be corrected.","section":"Appendix B"},{"comment":"The reference to Von Oswald et al. is rendered as 'V on Oswald' and should be fixed.","section":"References"},{"comment":"The table title says 'Pred. Length' but the text describes it as batch size; please clarify in the caption that the prediction sequence length is the number of objects used to compute the loss per gradient step.","section":"§3, Table 2"}],"recommendation":"major_revision","confidential_remarks":"I agree with the reader's conditional assessment. The empirical finetuning comparison is solid, reproducible, and publishable on its own; the mechanistic section, however, is not yet at the level claimed in the abstract. The main path to acceptance is either to add a causal intervention or to reframe the mechanism as a correlational observation. I would not reject, because the practical recommendation does not depend on the mechanism and the benchmark results are useful to the tabular DL community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The practical finding here is the real contribution: full finetuning, properly configured, beats PEFT and in-context baselines on medium tabular datasets, and the paper gives a recipe (tune LR, large prediction batches, early stopping) that practitioners can use. The comparison is systematic—multiple seeds, tuned baselines, honest reporting on TabReD where finetuned TabPFNv2 is unstable. The result that full finetuning beats partial finetuning contradicts v1 findings and is worth knowing.\n\nThe mechanism story—finetuning refines last-layer query-key dot products to better reflect target similarity—is plausible, and the diagnostics (attention-weighted kNN, entropy shifts, sample-wise error changes) are consistent with it. But the abstract's \"stems from\" overstates the evidence. Section 4 itself labels the retrieval mechanism a conjecture, and nothing in the paper intervenes on attention weights independently. Table 3's kNN proxy bypasses the model's actual output path: it averages raw training targets with attention weights, while TabPFNv2 uses those weights to combine value representations before a prediction head. So the table shows the attention weights encode better target similarity after finetuning, not that this similarity is what drives the model's own predictions. The stress-test note is exactly right on this.\n\nOther soft spots are minor. Table 3 is single-seed. Dataset selection is confined to what fits on one 80GB GPU, and the paper says so in Limitations. The code is promised but not verifiable from the preprint. None of these sink the practical claim, which is independent of the mechanism.\n\nWho this is for: anyone working on tabular foundation models or retrieval-based tabular deep learning. The finetuning protocol is a useful baseline for future work. The mechanism section needs revision—either soften the abstract and conclusion to match the conjecture, or add interventions (e.g., freeze attention weights, finetune only values/keys, or perturb attention scores). I would send this to peer review; the empirical contribution deserves referee time, and the mechanism overreach is fixable.","headline":"A useful, honest finetuning study for TabPFNv2 whose practical recipe holds up, but the mechanism claim is correlational and the abstract overstates it.","tokens_in":24250,"tokens_out":1881,"would_cite":true,"duration_ms":20264,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Finetuning improves the tabular foundation model TabPFNv2 by refining the query–key similarity scores in its last-layer attention, so that the model attends to the training rows whose labels actually matter.","keywords":["tabular foundation models","finetuning","in-context learning","attention mechanism","retrieval-based prediction","TabPFNv2","parameter-efficient finetuning","temporal shift"],"falsifier":"Take a finetuned TabPFNv2 and freeze everything except randomly perturbing the last-layer attention projections; if test accuracy barely changes while the attention weights are destroyed, the refined query-key dot products are not the causal channel. A cleaner check: replace the trained attention scores on test samples with the finetuned dot products computed from a model whose last layer has been reinitialized; if this recovers most of the finetuning improvement, the similarity signal carries the gain, and if it does not, the gain lives elsewhere.","tokens_in":23349,"feed_emoji":"🎯","tokens_out":9702,"duration_ms":79252,"temperature":0.7,"pith_summary":"This paper studies what happens when you take TabPFNv2, a tabular foundation model that makes predictions by attending to the training rows placed in its context, and adapt it with gradient-based finetuning. The authors establish that plain full finetuning is the most practical way to adapt the model on datasets up to about 50,000 rows, performing as well as or better than LoRA and other parameter-efficient alternatives while converging faster. They then identify why finetuning helps: after adaptation, the dot products between test-query and training-key representations in the last attention layer reflect true label similarity far more accurately, so the attention mechanism can weight the right in-context examples. Because of this, finetuned TabPFNv2 reaches state-of-the-art accuracy on standard academic benchmarks with I.I.D. splits, while on real-world datasets with temporal shift and heavy feature engineering it remains less stable than non-foundational baselines.","feed_headline":"Full finetuning wins for TabPFNv2 by honing its retrieval signal","feed_subtitle":"Sharper attention picks better training rows, lifting finetuned TabPFNv2 to state-of-the-art on academic benchmarks.","key_machinery":"The load-bearing object is the last-layer inter-sample attention of TabPFNv2, a tabular foundation model that predicts each test row by attending over the training rows provided in its input context. The query-key dot products in this attention layer act as a learned similarity measure between a test object and each in-context training object, and the attention softmax weights the corresponding training labels to form the prediction. The paper reads this mechanism as an implicit retrieval system, analogous to explicitly retrieval-based models, and validates that reading by using the attention weights alone as a weighted-$k$NN predictor: if the weights truly reflect target closeness, that predictor should be accurate. Finetuning is shown to improve the model precisely by sharpening these dot products, so the same attention machinery produces a better-matching set of neighbors.","core_discovery":"The central claim is that the benefit of finetuning TabPFNv2 can be traced to a single mechanism: the model's last-layer inter-sample attention, which is used to retrieve relevant training examples and weight their labels, produces query-key dot products that are more faithful to the true target similarity after finetuning. The paper supports this with a direct test: using only the attention weights as a weighted-nearest-neighbor predictor over training labels, the finetuned weights approximate test targets far better than the pretrained weights, and in most datasets the largest prediction improvements occur exactly on the test samples whose attention distribution becomes more concentrated after finetuning. In the authors' view, finetuning does not teach the model new features or a better output head; it repairs the similarity metric that the existing in-context retrieval logic relies on.","pith_inferences":["The paper's own evidence is correlational, so an intervention is the natural next test: if the query-key dot products are the causal channel, artificially sharpening or reweighting them at inference time should reproduce part of the finetuning gain without any gradient updates.","The finding that finetuning helps more on larger datasets suggests the pretrained similarity metric degrades relative to what the data can support; this predicts that the gap between finetuned and in-context TabPFNv2 should widen further as dataset size grows, up to the model's context limit.","The attention-entropy diagnostic could be turned into a practical tool: monitoring the shift in last-layer attention entropy on a validation set during finetuning may give an early signal of whether adaptation is actually improving the retrieval mechanism."],"forward_implications":["Full finetuning, with the right learning-rate schedule, should be the default adaptation method for TabPFNv2 on datasets up to tens of thousands of rows; LoRA and partial-update schemes add complexity without clear accuracy gains.","Because the gains come from the similarity signal, future attempts to improve TabPFNv2 adaptation can focus on the metric that drives the last-layer attention rather than on new feature embeddings or prediction heads.","On I.I.D. academic benchmarks, finetuned TabPFNv2, especially in ensembles, becomes competitive with or better than current non-foundational tabular deep-learning models.","On datasets with gradual temporal shift and rich feature sets, in-context and finetuned TabPFNv2 are less stable, and non-foundational methods like TabM retain an advantage."],"supporting_citations":[{"why":"Defines TabPFNv2, the model under study, and the in-context learning paradigm that this paper finetunes.","marker":"Hollmann et al., 2025"},{"why":"Prior finetuning study of the earlier TabPFN that argued partial finetuning prevents overfitting; the paper's full-finetuning result directly contradicts this.","marker":"Feuer et al., 2024"},{"why":"Another prior work recommending partial finetuning for tabular PFNs; it provides the contrast for the paper's main finetuning claim.","marker":"Xu et al., 2024"},{"why":"Introduces LoRA, the parameter-efficient finetuning baseline that full finetuning is compared against.","marker":"Hu et al., 2021"},{"why":"ModernNCA, the explicitly retrieval-based model whose per-dataset gains are highly correlated with TabPFNv2's, supporting the implicit-retrieval analogy.","marker":"Ye et al., 2024"},{"why":"TabR, another retrieval-based tabular model, anchors the analogy to explicit retrieval mechanisms.","marker":"Gorishniy et al., 2024"},{"why":"TabM, the state-of-the-art non-foundational baseline that finetuned TabPFNv2 must beat on academic and TabReD benchmarks.","marker":"Gorishniy et al., 2025"},{"why":"TabReD, the benchmark with temporal shift and rich features where finetuned TabPFNv2 remains less stable than prior methods.","marker":"Rubachev et al., 2025"}],"fun_headline_variants":["Finetuning TabPFNv2 sharpens its attention for better retrieval","TabPFNv2 finetuning: attention weights become truthful similarity","Why finetuning TabPFNv2 works: it fixes retrieval attention","Finetuned TabPFNv2's attention picks more relevant rows","TabPFNv2 finetuning improves retrieval, not features"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The diagnosis assumes that TabPFNv2's predictions can be read off from its last-layer attention weights, so that changes in those weights explain the performance gains; the paper supports this with correlations and weight-based proxy predictions, not by intervening on the attention mechanism directly.","fun_headline_variants_meta":{"raw":{"variants":["Finetuning TabPFNv2 sharpens its attention for better retrieval","TabPFNv2 finetuning: attention weights become truthful similarity","Why finetuning TabPFNv2 works: it fixes retrieval attention","Finetuned TabPFNv2's attention picks more relevant rows","TabPFNv2 finetuning improves retrieval, not features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000319,"raw_usage":{"total_tokens":1847,"prompt_tokens":1040,"completion_tokens":807,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":656,"completion_tokens_details":{"reasoning_tokens":710}},"tokens_in":656,"tokens_out":807,"duration_ms":8557,"temperature":1.0,"reasoning_tokens":710,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:57:07.813478+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a finetuned TabPFNv2 and freeze everything except randomly perturbing the last-layer attention projections; if test accuracy barely changes while the attention weights are destroyed, the refined query-key dot products are not the causal channel. A cleaner check: replace the trained attention scores on test samples with the finetuned dot products computed from a model whose last layer has been reinitialized; if this recovers most of the finetuning improvement, the similarity signal carries the gain, and if it does not, the gain lives elsewhere.","supporting_citations":[],"review_version":1}