{"id":"783f1083-ac84-49ff-9979-3c239e8e8baf","arxiv_id":"2605.18147","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Tabular foundation models, used zero-shot, match or beat tuned gradient boosting on average in credit PD and LGD benchmarks, with a larger edge on small datasets.","lead":"This benchmark compares pretrained tabular foundation models against tuned gradient boosting, deep learning, and classical methods for predicting loan default (PD) and loss given default (LGD) on 21 credit datasets. It finds foundation models rank first on average, with the largest recorded edge on small datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Small-data advantage (Abstract/§5.4) is a correlation across heterogeneous datasets, not a causal size effect; confounds (dimensionality, default rate, origin) could drive the result.","rationale":"Both the reader and I focus on §5.4: the small-data advantage is the claim that would make foundation models a 'game changer' for credit risk, and it rests on correlational evidence that does not rule out confounds. The first component of the central claim (TFMs generally perform best) is also weakened by the lack of significant pairwise differences against top baselines (0/231 for LGD), but that weakness is less damaging: even if TFMs are only statistically tied with CatBoost/GBMs, they remain competitive out-of-the-box, and the paper's recommendation could be softened. The small-data claim, by contrast, is the differentiator and the most actionable for practitioners (SME lending, low-default portfolios). If the size effect is confounded, the abstract's 'significant improvement ... as dataset size shrinks' is unsupported. The proposed check—partial correlation controlling for dimensionality, default rate, and origin—directly tests causality using data the authors already have. I therefore recommend no change to the CONDITIONAL verdict: the paper is a useful benchmark but should not be accepted without addressing the confound and ideally adding more LGD foundation models, as the reader suggested.","tokens_in":53037,"tokens_out":4975,"duration_ms":51042,"concrete_test":"Recompute the size analysis from §5.4 using a partial Spearman correlation or a linear mixed model on the 14 PD datasets (and separately the 7 LGD datasets): predict the per-dataset TFM rank advantage (e.g., rank(TabICL)-rank(CatBoost) for PD; rank(TabPFNv2)-rank(CatBoost) for LGD) from log10(n) plus covariates log10(p), default rate (PD), and source (public/proprietary). If the adjusted effect of log10(n) is not significant at α=0.05 or changes sign, the small-data claim in the Abstract/§5.4 is not supported. As a robustness check, repeat using only datasets with n<5,000; if the TFM advantage is not consistently positive there, the 'small portfolio' transfer fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.4 claims TFMs 'offer significant improvement in predictive performance when the size of the dataset ... grows smaller' (Abstract). The evidence is (i) Spearman correlations between per-dataset method rank and dataset size across 14 PD and 7 LGD datasets (Figures 8-9) and (ii) learning curves obtained by subsampling large datasets to 500-15,000 rows (Figures 10-11). Both designs assume dataset size is the causal driver. But the dataset collection confounds size with other attributes: the smallest PD datasets (German Credit, Thomas) have high/default rates and low dimensionality, while the largest (Hackerearth, PD6) are high-volume Kaggle tasks; LGD small datasets (Base Model/Base Modelisation) have >200 features, whereas the large Freddie Mac/HELOC have 20/8 features. A Spearman correlation across these points cannot separate size from feature count, default rate, or data origin/quality. The learning-curve experiment removes between-dataset confounds but creates 'small data' by random row subsampling, which preserves the feature space and default incidence of large retail datasets; it does not simulate SME/low-default portfolios where data are scarce in a different way (e.g., few defaults, missing fields, different borrower types). The unexplained decline of TabPFNv2's LGD learning curve after 8,000 rows (Fig. 11) further signals that the size-response is not a stable, monotonic TFM advantage. Because the small-data benefit is the paper's headline practical contribution, the causal reading is load-bearing; if the size effect disappears under covariate adjustment, the central claim is reduced to 'TFMs are competitive on average,' which would not warrant the 'Game Changer' framing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks tabular foundation models (TFMs) against a broad panel of classical, tree-based, and deep learning methods for two credit-risk tasks: probability of default (PD) classification and loss given default (LGD) regression. Using 14 PD and 7 LGD datasets, five-fold cross-validation, and a wide range of metrics, it reports that TabICL achieves the highest average AUC and the highest PD fold-level win rate (PAMA 25.7%), while TabPFNv2 achieves the highest average R² and wins 45.7% of LGD folds. Friedman tests reject global equivalence of methods, but only 22/406 PD and 0/231 LGD pairwise comparisons are significant after Holm correction. The paper further claims that TFMs offer particularly large gains in small-data settings, based on Spearman correlations between method rank and dataset size and on learning curves obtained by subsampling large datasets. The conclusion is that TFMs generally outperform state-of-the-art alternatives and are especially attractive for small credit portfolios.","tokens_in":53323,"tokens_out":6399,"duration_ms":68330,"significance":"The study is a broad, mostly self-contained empirical benchmark in an area where independent evidence is scarce. Its strengths include the use of a unified preprocessing/evaluation framework (TALENT), out-of-the-box evaluation of TFMs against tuned baselines, multiple performance metrics, and standard statistical procedures (Friedman, Holm-corrected Wilcoxon, PAMA). If the headline claims held, the paper would be a valuable reference for credit-risk modelers and would lend support to the 'foundation models are a game changer' narrative. As it stands, the two central claims—'generally perform best' and 'significant improvement as dataset size shrinks'—outrun the evidence: pairwise significance is weak, especially for LGD, and the small-data analysis is confounded. With appropriate revisions and more cautious wording, the paper could still make a useful contribution.","major_comments":[{"comment":"The headline claim that TFMs 'offer significant improvement in predictive performance when the size of the dataset ... grows smaller' is not established. The Spearman correlations in Figs. 8–9 are computed across heterogeneous datasets without adjusting for confounds: e.g., small PD datasets (PD8, PD10) have 20/14 features and high default rates, whereas the large PD14 has 2,986 features; among LGD datasets, the small Base Model/Base Modelisation have 202/256 features while the large HELOC has 8. The learning-curve experiment removes between-dataset confounds only by random row subsampling, which preserves feature spaces and default incidence; it does not reproduce the structure of SME/low-default portfolios. Moreover, Fig. 10 appears to compare TFMs mainly against XGBoost and logistic regression, not the strongest GBM baselines named in the text (CatBoost, LightGBM), and Fig. 11 shows a","section":"§5.4, Figs. 8–11; Abstract"},{"comment":"The conclusion that TFMs 'generally perform best across datasets and tasks' is based on average metrics and PAMA, but the pairwise statistical evidence is weak: only 22/406 PD comparisons and 0/231 LGD comparisons are significant after Holm correction. PAMA is descriptive, and the Friedman test only establishes that some global differences exist, not that the top method is significantly better than the runner-up. Given that the best TFM and the best GBM differ by about 0.0023 AUC in PD and 0.0054 R² in LGD, the word 'best' is an overstatement. Please report effect sizes/confidence intervals for the top pairs and either temper the claim to 'competitive on average' or provide a more powerful analysis, especially for LGD where no pairwise difference survives correction.","section":"§5.3; Tables 4–5; Abstract"},{"comment":"The LGD component of the benchmark evaluates only one foundation model (TabPFNv2), yet the paper's abstract and conclusion speak of 'tabular foundation models generally perform best' across tasks. A single model is a thin basis for the 'paradigm shift' claim in regression. The average R² gap is also very small (0.4455 vs. 0.4401 for CatBoost), so the practical significance is unclear without per-dataset breakdowns. Please either add more regression-capable TFMs or restrict the generalizing claim to the particular model evaluated.","section":"§5.2, Table 5"}],"minor_comments":[{"comment":"In the supplied manuscript, these figures appear as unreadable sequences of '/uni...' glyphs. If this is a rendering artifact, the final version must show actual axes, labels, and legends; otherwise the visual evidence cannot be inspected.","section":"Figures 2–7"},{"comment":"The text says the learning curves include CatBoost, FTT, LightGBM, and XGBoost, but the figure captions list only TabICL/TabPFN/Logistic Regression/XGBoost and TabPFNv2/Linear Regression/XGBoost. Please clarify which curves are actually plotted and why the strongest GBM baselines are absent.","section":"§5.4, Figs. 10–11 captions"},{"comment":"The abstract emphasizes that TFMs are used 'out-of-the-box, without hyperparameter tuning', while §4.3 describes hyperparameter optimization for the benchmarked methods. Please clarify explicitly that HPO applies only to non-TFM baselines, so readers do not infer an asymmetric protocol.","section":"§4.3"},{"comment":"No code or data availability statement is included. For a benchmark paper of this scope, a reproducibility appendix or repository link would substantially increase its value.","section":"General"},{"comment":"Several references are duplicated (e.g., Shwartz-Ziv and Armon 2022; Hurlin et al. 2018) and some entries have inconsistent formatting. A careful reference cleanup is needed.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid benchmark but its two headline claims outrun the evidence. The small-data section, in particular, is the practical payoff of the paper and needs either a confound-controlled analysis or a clearly hedged claim. The authors seem aware of some limitations (e.g., the unexplained TabPFNv2 decline), which makes revision feasible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth your time, but read the abstract as hype and the body as a competent benchmark. The genuinely new thing is the first systematic credit-risk evaluation of tabular foundation models (TabPFN, TabPFNv2, TabICL, MITRA, etc.) against tuned GBMs and deep tabular learners, on 14 PD and 7 LGD datasets. The core average-result is credible: TabICL edges CatBoost on PD AUC (0.7517 vs 0.7494) and TabPFNv2 edges CatBoost on LGD R² (0.4455 vs 0.4401). These are small absolute differences, but the result is not an artifact of the setup. They use standard metrics, five-fold CV, per-fold HPO, and the PFNs are zero-shot, which makes the comparison fair in the sense that the baselines are tuning-adjusted. The statistical section is honest: Friedman is significant, but pairwise Holm-corrected tests yield only 22/406 significant PD comparisons and 0/231 for LGD. They explicitly note the power problem. That transparency earns credit.\n\nThe soft spot is exactly what the stress-test flags: the small-data conclusion in §5.4. The abstract says TFMs 'offer significant improvement' as dataset size shrinks, but the evidence is a Spearman correlation across heterogeneous datasets where size is confounded with dimensionality, default rate, and origin, plus learning curves built by random subsampling of large retail datasets. Randomly removing rows from a 150k-row Kaggle set does not reproduce a 1,000-row low-default portfolio with a few dozen defaults and messy missingness. The LGD learning curve even shows TabPFNv2 declining after 8,000 rows, which undercuts any clean monotonic story. If the size effect is load-bearing for the 'Game Changer' title, it is not established. The reader's conditional verdict is right. The fix is straightforward: add per-fold uncertainty/error bars, run a covariate-adjusted test or an interaction model with a formal size-by-method interaction, include more than one LGD foundation model, and release the pipeline. Lack of code/data also prevents independent verification, though the benchmark is self-contained and not circular.\n\nWho's it for? Credit-scoring researchers and anyone assessing tabular foundation models in high-stakes domains. It deserves a serious referee, but the referee should demand the small-data analysis be reframed as suggestive, not confirmatory. I'd accept for peer review with the expectation of major revision.","headline":"Solid first credit-risk benchmark of tabular foundation models; the average-performance claim holds, but the headline small-data advantage is not statistically established.","tokens_in":53958,"tokens_out":3048,"would_cite":true,"duration_ms":28879,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T05","91G40"],"pacs":[],"model":"deepseek-v4-flash","headline":"Pretrained tabular foundation models, used with zero task-specific training, match or beat tuned gradient-boosting and deep-learning rivals on both probability-of-default and loss-given-default benchmarks, and their relative advantage grows","keywords":["credit risk modeling","probability of default","loss given default","tabular foundation models","prior-data fitted networks","zero-shot learning","benchmarking","small-data learning"],"falsifier":"A controlled subsampling experiment on one large retail default dataset: take the same portfolio, draw training sets of 500, 1,000, 2,000, 5,000, and 10,000 observations, and compare TabICL/TabPFNv2 against tuned boosting baselines. If the tuned baselines match or exceed the foundation models at small sizes once dimensionality, default rate, and feature semantics are held fixed, the paper's small-data claim is refuted. For LGD, the same test should include at least two or three zero-shot regressors, since the paper evaluates only one.","tokens_in":52876,"feed_emoji":"📊","tokens_out":6685,"duration_ms":61042,"temperature":0.7,"pith_summary":"This paper asks whether a new family of models—tabular foundation models, pretrained on synthetic data and applied without task-specific training—can beat the tuned machine-learning pipelines that currently set the standard in credit-risk practice. Across 14 probability-of-default datasets and 7 loss-given-default datasets, it finds that these zero-shot models generally rank first: TabICL leads default prediction on average AUC and wins 25.7% of folds, while TabPFNv2 leads loss prediction on R² and wins 45.7% of folds. The paper also claims that these models improve most relative to competitors as datasets get smaller, which matters for SME lending, low-default portfolios, and new products. If true, lenders could deploy strong models faster and with less computational overhead, while keeping a consistent method across PD and LGD.","feed_headline":"Tabular AI models top credit-risk benchmarks with zero tuning","feed_subtitle":"Zero-shot models match or beat tuned boosting on default and loss prediction—and their edge grows as portfolios shrink.","key_machinery":"The load-bearing mechanism is the prior-data fitted network (PFN): a transformer pretrained on large numbers of synthetic tabular datasets sampled from priors over data-generating processes, so that inference on a new dataset is a single forward pass over labeled and unlabeled rows—no gradient updates, no tuning. The paper also relies on TabPFNv2's regression variant, which discretizes the continuous LGD target into an ordinal piecewise-constant distribution, a design well suited to the zero-one-inflated, bimodal loss distributions in the data. The evaluation machinery is a standardized cross-validation protocol with hyperparameter optimization for all non-PFN baselines, plus PAMA win rates,","core_discovery":"The paper's central claim is empirical: in a five-fold cross-validation benchmark spanning 14 PD and 7 LGD datasets, tabular foundation models generally outperform state-of-the-art alternatives even when used out-of-the-box with no hyperparameter tuning. TabICL achieves the highest average AUC (0.7517) and the best fold-level win rate (25.7%) for PD, with foundation models collectively top in 44.3% of folds; TabPFNv2 achieves the highest average R² (0.4455) and wins 45.7% of folds for LGD. A global non-parametric ranking test rejects the null of equal performance, though pairwise post-hoc significance is limited—especially for LGD, where only seven datasets make multiple-comparison-corrected","pith_inferences":["Editorial extension: because the LGD leg rests on a single foundation model (TabPFNv2), the strongest testable extension is to run additional zero-shot regressors and standard LGD econometric baselines (e.g., two-part and beta models) on the same seven datasets; if the lead survives, the regression-side claim is much firmer.","Editorial extension: the paper's own learning curves show TabPFNv2's R² declining as training rows pass about 8,000, with the authors unsure why; a focused study of whether the pretrained prior becomes a liability on larger in-domain samples, and whether ensembling a PFN with a GBM recovers the loss, would sharpen the small-data thesis.","Editorial extension: a broader reading is that if zero-shot tabular models keep improving, the default workflow could shift from tuning a GBM per portfolio to validating a foundation model out of the box—but calibration, feature attribution, and stability under economic regime shifts would need dedicated study before that shift is safe.","Editorial extension: the small-data result could be tested more cleanly by holding dataset origin constant—subsampling a single large retail portfolio to many sizes and comparing methods—which would separate the size effect from confounds like default rate and dimensionality."],"forward_implications":["For probability-of-default modeling, a zero-shot model (TabICL) can match or beat tuned gradient boosting on discrimination, so a lender could ship a competitive scorecard without per-portfolio hyperparameter search.","For loss-given-default, TabPFNv2 shows that framing regression as discretized ordinal classification can handle the bimodal, zero-one-inflated loss distribution better than tuned tree ensembles and deep regressors on these data.","In small-data settings—SME lending, low-default portfolios, new products—the relative advantage of foundation models is largest, suggesting they are most useful where conventional models have the least signal.","Operationally, removing tuning and retraining cuts time-to-model and compute, and allows one consistent model family across PD and LGD, which could simplify validation and monitoring.","The global statistical test rejects equal performance among the 29 PD / 22 LGD methods, but pairwise significance is scarce; the practical ranking of the top methods should be treated as suggestive rather than definitive per-pair proof."],"fun_headline_variants":["Tabular foundation models outperform tuned baselines on credit risk","No tuning needed: tabular foundation models lead credit-risk benchmarks","Foundation models beat gradient boosting in credit risk without tuning","Pretrained tabular models top credit-risk tests out-of-the-box"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The small-data conclusion stands only if dataset size itself, not some correlated property of the small datasets (different loan products, feature counts, or default rates), is what drives the foundation models' relative advantage.","fun_headline_variants_meta":{"raw":{"variants":["Tabular foundation models outperform tuned baselines on credit risk","No tuning needed: tabular foundation models lead credit-risk benchmarks","Foundation models beat gradient boosting in credit risk without tuning","Pretrained tabular models top credit-risk tests out-of-the-box"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000239,"raw_usage":{"total_tokens":1380,"prompt_tokens":805,"completion_tokens":575,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":515}},"tokens_in":549,"tokens_out":575,"duration_ms":5605,"temperature":1.0,"reasoning_tokens":515,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T13:42:15.100014+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled subsampling experiment on one large retail default dataset: take the same portfolio, draw training sets of 500, 1,000, 2,000, 5,000, and 10,000 observations, and compare TabICL/TabPFNv2 against tuned boosting baselines. If the tuned baselines match or exceed the foundation models at small sizes once dimensionality, default rate, and feature semantics are held fixed, the paper's small-data claim is refuted. For LGD, the same test should include at least two or three zero-shot regressors, since the paper evaluates only one.","supporting_citations":[],"review_version":2}