{"id":"19c2a62a-6b8f-4e81-a344-3b18e7bc609c","arxiv_id":"2502.02672","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Seeding gradient boosted trees with scaled transformer scores (LLM-Boost and PFN-Boost) improves average tabular classification performance over either component alone for most tested sample sizes.","lead":"This paper proposes a simple way to combine transformer models with gradient-boosted decision trees: use the transformer's prediction scores as a starting point and let the trees learn the remaining error. The resulting methods, LLM-Boost and PFN-Boost, often beat either model alone on small and medium tabular datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Scaling parameter tuned on 10–25 validation rows can overfit; reported small-sample gains may be selection artifacts.","rationale":"The reader's weakest assumption identifies exactly the load-bearing condition: the scaling parameter must be tuned on a very small validation split, and the central claim is measured on test after that noisy selection. This is the most critical soft spot because it directly affects the validity of the empirical evidence for the headline claim. If the selected s is overfit to validation noise, then the reported advantage of the boosted methods over baselines could be an artifact of having an extra continuous tuning parameter that can fit the validation set more closely than the binary Selection baseline. The concrete test would settle this by comparing against fixed-s and nested-validation variants, which remove the overfitting channel. The paper's other issues, such as metric inconsistency and missing significance tests, are secondary because they are presentation/statistical weaknesses rather than a specific mechanism that could invalidate the central finding. The concern is correctable and does not necessarily sink the core idea, so the reader's CONDITIONAL verdict remains appropriate; I would not change it without additional evidence.","tokens_in":36677,"tokens_out":9291,"duration_ms":93423,"concrete_test":"For datasets with train/val size at most 50, recompute PFN-Boost and LLM-Boost test AUC under three protocols: (a) the paper's Optuna selection of s on the validation split, (b) a fixed s=1, and (c) s chosen via a nested validation split held out from the current validation set. Compare average test AUC against the best standalone component and the Selection baseline. If (b) or (c) does not consistently beat the standalone, the reported gains are an artifact of overfitting s to a tiny validation set. Also compute paired bootstrap confidence intervals for the differences at each sample size.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LLM-Boost and PFN-Boost outperform both standalone components across sample sizes rests on the scaling parameter s being chosen reliably. Section 3.3 and Appendix A specify tuning s over [0, 1e4] with 30 Optuna trials on a validation split that is as small as 10 rows (e.g., Abalone with train=10, val=10 in Table 3). Since s=0 is in the search space, the procedure is guaranteed to match or beat the GBDT on validation, turning the tuning into an aggressive model-selection step on a tiny sample. The test AUC then reflects the selected s, but that selection may not transfer. The 'Selection' baseline makes only a binary choice between GBDT and transformer on the same validation, while the continuous s can more closely fit validation noise, giving the boosted methods an unfair advantage in the comparison. The reported small-sample gains, and the claim that PFN-Boost is best for all but very small sizes, may therefore be artifacts of this validation overfitting rather than genuine fusion benefits.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a simple fusion mechanism, LLM-Boost and PFN-Boost, in which the raw prediction scores of a transformer (an LLM or TabPFN) are scaled by a tunable parameter and used as the initial prediction of a gradient-boosted decision tree, which then learns residuals from the transformer scores. Experiments on 16 tabular classification datasets across sample sizes 10 to full dataset compare these methods against the standalone GBDT, standalone transformer, validation-based selection, stacking, TabLLM, and AutoGluon. The central claims are that LLM-Boost and PFN-Boost outperform both of their constituent models on intermediate dataset sizes, and that PFN-Boost achieves the best average performance among all tested methods for all but the very smallest sample sizes.","tokens_in":36853,"tokens_out":3459,"duration_ms":33868,"significance":"If the central claims hold, the proposed methods offer a lightweight, no-fine-tuning way to inject pretrained transformer priors and natural-language column-header information into scalable GBDT pipelines, which is practically relevant for small and medium tabular classification problems. The paper's strengths include its breadth (16 datasets, multiple sample sizes, multiple LLM/GBDT combinations), the release of code, an ablation of column-header shuffling, and a comparison with TabLLM and AutoGluon. However, the evidence as presented has important gaps: the scaling parameter is tuned on extremely small validation splits, the evaluation mixes AUC and accuracy metrics when constructing aggregate curves, and no significance tests or error bars for the TabPFN baseline are provided for the often small performance differences.","major_comments":[{"comment":"The scaling parameter s is tuned with 30 Optuna trials on validation sets as small as 10 rows, and the search space includes s=0 (Appendix A). Because s=0 reproduces the standalone GBDT, the tuning procedure is guaranteed to match or improve validation performance by construction, but this does not guarantee test improvement. With 10 validation points, the selected s can overfit noise, so the reported test gains may partly reflect the specific validation-split realization rather than a genuine fusion benefit. The authors should report the distribution of selected s values, evaluate sensitivity to validation size (e.g., by varying the validation split or using repeated cross-fitting), and compare against a selection rule with equivalent capacity (e.g., picking s from the same search grid with the same number of trials) to support the claim that the gains come from the fusion mechanism rather than from overfitting the validation set.","section":"Section 4.2, Table 3 (rows with train/val = 10/10)"},{"comment":"Table 3 reports AUC for PFN-Boost, while Table 4 reports accuracy for LLM-Boost; nevertheless, Section 5 states that rank and z-score are calculated 'based on AUC' and Figures 3 and 4 plot 'Average AUC' for both methods. Mixing accuracy and AUC across methods and then averaging them as a single 'AUC' metric is statistically invalid and makes the aggregate comparisons misleading. The authors should either compute AUC for all experiments (including LLM-Boost) or clearly separate accuracy-based and AUC-based analyses and adjust the wording of the claims accordingly.","section":"Section 5.1, Section 5.2, Tables 3 and 4"},{"comment":"TabPFN results are reported without standard errors, while the other methods have ± errors, and no significance tests are provided for the often tiny differences between PFN-Boost and TabPFN or XGBoost (e.g., Abalone train=10: PFN-Boost 0.7119±0.0309 vs TabPFN 0.7109; Churn train=10: PFN-Boost 0.7122±0.0000 vs XGB 0.7122±0.0000). The claim that PFN-Boost 'achieves the best average performance among all methods we test for all but very small dataset sizes' requires either error bars for all methods and paired significance tests across seeds, or a clear statement that the observed differences are not statistically significant. Without this, the headline claim is not yet supported by the presented evidence.","section":"Section 5.1, Table 3"}],"minor_comments":[{"comment":"The caption reads 'An few-shot prompt'; it should read 'A few-shot prompt'.","section":"Figure 2 caption"},{"comment":"The caption uses 'negligent' where 'negligible' is meant: 'the seed values in LLM-Boost are negligent' should be 'negligible'.","section":"Figure 14 caption"},{"comment":"The sentence 'We use separate validation folds so that test data is new used for HPO trials' contains a typo; it should read 'test data is never used for HPO trials'.","section":"Section 4.2"},{"comment":"The text 'sample HPO initilization' should be 'sample HPO initialization'.","section":"Appendix B"},{"comment":"The notation pred(0,i) and pred(1,i) is not fully defined; in particular, it is unclear whether the index i is zero-based or one-based and how the constant C is chosen. A precise definition of the prediction interval would improve clarity.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a simple and potentially useful idea, and the experimental scope is commendable. However, the evaluation currently mixes AUC and accuracy in aggregate metrics and lacks statistical support for the headline claims. The scaling-parameter tuning issue on tiny validation sets is a legitimate concern that the authors should address with additional analyses. I would encourage the editor to seek a revision that clarifies these points rather than rejecting the paper outright."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is genuinely simple and useful: replace the constant offset in a GBDT with scaled transformer logits, then fit the trees to the residuals. I haven't seen that exact move in prior work, and the paper makes a fair case that it can beat both components on intermediate sample sizes. The evaluation is broad – 16 datasets, multiple sample sizes, several transformer/GBDT combinations, and ablations on model size and shots – plus they release code. That's real value.\n\nThe soft spots are real but not disqualifying. The tables mix AUC and accuracy without clear labels; TabPFN results lack error bars; and many of the reported gains are tiny fractions of a percent with overlapping standard errors. No significance tests are reported, so some of the \"outperforms everything\" language in the abstract is stronger than the evidence. The stress-test concern about the scaling parameter also holds up: tuning s on 10–25 validation rows with 30 Optuna trials, with s=0 in the search space, guarantees a validation win and can overfit noise. The Selection baseline gets only a binary choice, so the comparison at small n is not apples-to-apples. This doesn't kill the method – the idea stands and the full-dataset results are sensible – but the small-sample gains should be presented as suggestive rather than conclusive, and the claim that PFN-Boost is best for all but very small sizes needs hedging.\n\nWho gets value from this? Practitioners working with tabular data between 50 and a few hundred rows, and researchers studying ensembling or transfer for tabular ML. The paper is worth a serious referee: the method is cheap, reproducible, and likely to be cited. It needs revision – fix the metric tables, add error bars for TabPFN, run significance tests or at least paired comparisons, and address the validation-overfitting risk with a nested or larger validation scheme. I'd send it to review with those requests.","headline":"A simple, plausible fusion of transformer scores and GBDTs; the experiments are broad but the small-sample gains may partly be validation-tuning artifacts, so the headline needs softening.","tokens_in":37409,"tokens_out":1460,"would_cite":true,"duration_ms":19371,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Seeding gradient-boosted trees with a transformer's logits, so the trees learn the residuals, lets tabular classifiers beat either model alone across most dataset sizes.","keywords":["tabular data","gradient-boosted decision trees","large language models","TabPFN","boosting","few-shot classification","ensemble methods","residual learning"],"falsifier":"A reader could take the same 16 datasets, fix a large test split, and run PFN-Boost twice: once with the scaling parameter tuned by 30 hyperparameter trials on a 10-row validation split and once with the scaling parameter fixed to zero. If the tuned version does not beat the zero version on most datasets, the reported small-sample advantage is an artifact of tuning on a tiny validation set.","tokens_in":36497,"feed_emoji":"📊","tokens_out":6057,"duration_ms":51383,"temperature":0.7,"pith_summary":"The paper tries to establish that transformer models and gradient-boosted decision trees are not rivals to choose between, but components of one pipeline. It proposes replacing the constant starting point of a GBDT with the scaled logits of an LLM (LLM-Boost) or of TabPFN (PFN-Boost), so that the trees learn the residual error of the transformer. If true, practitioners gain the small-data, header-aware strength of transformers and the scalability of GBDTs at the same time, with no fine-tuning of the transformer. The headline finding is that PFN-Boost achieves the best average performance among all tested methods for all but very small dataset sizes, while LLM-Boost is strongest in the very small regime.","feed_headline":"PFN-Boost tops every tested model on tabular data","feed_subtitle":"Seeding gradient-boosted trees with TabPFN's logits beats both standalone models across most dataset sizes.","key_machinery":"The carrying mechanism is the replacement of the GBDT's constant initial prediction with scaled transformer logits: a single scaling parameter $s$ multiplies the transformer scores before they enter the ensemble, and every subsequent tree is fit to the residual of the combined prediction. Setting $s=0$ recovers the plain GBDT and sending $s$ large recovers the transformer, so tuning $s$ on a validation split interpolates between the two. For LLM-Boost the scores are extracted as the negative mean per-token cross-entropy of each verbalized class label; for PFN-Boost they are TabPFN's unnormalized output scores. Because the transformer scores are computed once and reused, the extra training cost beyond a standard GBDT pipeline is small.","core_discovery":"The paper's central claim is that a gradient-boosted decision tree can be seeded with a pretrained transformer's raw class scores, and that the tree ensemble then learns the residuals of those transformer predictions, giving a model that inherits both the transformer's prior and the tree's scalability. The authors instantiate this as LLM-Boost, which seeds XGBoost with the negative mean per-token cross-entropy scores of an instruction-tuned LLM prompted with few-shot serialized rows, and PFN-Boost, which seeds XGBoost with TabPFN's raw scores. On 16 classification datasets at train sizes from 10 to 500 plus full datasets, they report that both boosted methods match or outperform each standalone component at the extremes and outperform both in between, that LLM-Boost is best in the very small regime, and that PFN-Boost attains the best average performance among all tested methods for all but very small dataset sizes, without fine-tuning.","pith_inferences":["A direct corollary the paper does not test: the same residual-seeding trick should extend to other base learners and to regression, since nothing in the mechanism is specific to classification or to trees.","The method implies that the value of column headers can be quantified and traded off; the ablation shows headers matter most at small sample sizes, suggesting header quality, rather than dataset size alone, should determine whether to use an LLM seed.","A testable extension is to make the scaling parameter a function of dataset size or validation performance, or to select it by repeated cross-validation, which may stabilize the very-small-sample regime where tuning $s$ on 10 rows is noisy.","The PFN-Boost result suggests TabPFN's prior is strong enough that even at 500-1000 training rows it beats a from-scratch GBDT after residual correction, raising the question of whether larger-context tabular transformers could push the crossover point to even bigger datasets."],"forward_implications":["PFN-Boost gives the best average AUC among all methods tested for every sample size except the smallest, so practitioners with datasets too large for TabPFN alone can still use TabPFN's pretraining through a cheap GBDT wrapper.","LLM-Boost is strongest at 10-100 training samples, beating the LLM alone, XGBoost alone, selection, and stacking, and it outperforms TabLLM's fine-tuning baseline in most sample sizes without any fine-tuning.","The fusion is model-agnostic: swapping XGBoost for LightGBM or swapping the primary LLM for Flan-T5-XXL or Llama-3-8B-Instruct still yields gains over baselines, with the size of the gain tracking the strength of the transformer.","Column headers carry measurable value: shuffling the headers degrades LLM-Boost at small sample sizes, confirming that the LLM's semantic reading of headers, not just its pretraining, is part of what the trees inherit."],"supporting_citations":[{"why":"Supplies the pretrained tabular transformer whose raw scores seed PFN-Boost and defines the small-data baseline it must beat.","marker":"[19]"},{"why":"Provides the XGBoost implementation used as the base model and principal baseline for LLM-Boost and PFN-Boost.","marker":"[6]"},{"why":"Contributes the TabLLM few-shot LLM tabular baseline and the TabLLM dataset suite used in the comparison.","marker":"[18]"},{"why":"Provides the prompt-generation and serialization tools used to turn table rows into LLM prompts.","marker":"[28]"},{"why":"Supplies the AutoGluon automated ensembling baseline that LLM-Boost and PFN-Boost are compared against.","marker":"[11]"},{"why":"Supplies the hyperparameter-optimization routine used to tune GBDT hyperparameters and the scaling parameter.","marker":"[2]"},{"why":"Provides the instruction-tuned LLM whose scores seed the main LLM-Boost experiments.","marker":"[31]"},{"why":"Provides the Flan-T5-XXL LLM used in ablations and in the TabLLM and AutoGluon comparisons.","marker":"[8]"},{"why":"Provides the smaller Llama-3-8B-Instruct model used to test how LLM-Boost degrades with a weaker seed model.","marker":"[1]"},{"why":"Provides the LightGBM alternative GBDT used to show the boosting mechanism is not specific to XGBoost.","marker":"[22]"}],"fun_headline_variants":["PFN-Boost: XGBoost seeded with TabPFN wins tabular average","Transformer scores seed trees, beat both on many sizes","Tree-Transformer fusion tops tabular test, no fine-tuning","PFN-Boost beats TabPFN and XGBoost on mid-size tables"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a scaling parameter chosen on a tiny validation split of 10 to 50 rows will still be the right scale for the held-out test set, rather than just fitting noise in those few validation rows.","fun_headline_variants_meta":{"raw":{"variants":["PFN-Boost: XGBoost seeded with TabPFN wins tabular average","Transformer scores seed trees, beat both on many sizes","Tree-Transformer fusion tops tabular test, no fine-tuning","PFN-Boost beats TabPFN and XGBoost on mid-size tables"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1434,"prompt_tokens":1061,"completion_tokens":373,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":677,"completion_tokens_details":{"reasoning_tokens":291}},"tokens_in":677,"tokens_out":373,"duration_ms":4212,"temperature":1.0,"reasoning_tokens":291,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T11:30:49.844748+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could take the same 16 datasets, fix a large test split, and run PFN-Boost twice: once with the scaling parameter tuned by 30 hyperparameter trials on a 10-row validation split and once with the scaling parameter fixed to zero. If the tuned version does not beat the zero version on most datasets, the reported small-sample advantage is an artifact of tuning on a tiny validation set.","supporting_citations":[{"cited_title":"TabPFN: A transformer that solves small tabular classification problems in a second","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained tabular transformer whose raw scores seed PFN-Boost and defines the small-data baseline it must beat."},{"cited_title":"Tablet: Learning from instructions for tabular data.arXiv, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the prompt-generation and serialization tools used to turn table rows into LLM prompts."},{"cited_title":"Qwen2.5: A party of foundation models, September 2024","cited_arxiv_id":null,"evidence_quote":"Provides the instruction-tuned LLM whose scores seed the main LLM-Boost experiments."},{"cited_title":"Llama 3 model card, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the smaller Llama-3-8B-Instruct model used to test how LLM-Boost degrades with a weaker seed model."}],"review_version":1}