{"id":"505b981d-f072-496f-89fd-bfc790fb3b6d","arxiv_id":"2508.16355","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Pretraining a permutation-invariant quantile network across 101 tabular datasets improves fine-tuned accuracy and calibration, but the headline claim of beating well-tuned tree ensembles is contradicted by the paper's own baseline sweeps.","lead":"This paper introduces NIAQUE, a neural network that predicts any quantile of a regression target and is pretrained across 101 tabular datasets before fine-tuning on new tasks, reporting gains in low-data regimes and strong Kaggle results against TabPFN and TabDPT. A generalist should read it for the claim that transfer learning can make neural networks competitive with gradient-boosted trees on tabular data, a long-standing open question.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Table 1 claim that NIAQUE-Global 'significantly outperforms' tree baselines is not supported by the paper's own hyperparameter sweeps: with best appendix settings, XGBoost/CatBoost/LightGBM beat it on SMAPE and RMSE.","rationale":"Good-faith read: the transfer-learning contribution is plausible and directly tested — Table 2 shows pretraining consistently beats scratch in the held-out split, and the Kaggle Abalone experiment includes NIAQUE-Pretrain-100 without the original Abalone data, so the transfer result does not rest on leakage. The weakest point is the comparison claim. The reader's weakest_assumption matches what I see: Table 1 reports tree baselines at settings that the paper's own hyperparameter sweeps show to be suboptimal. I checked the appendices; the margins are decisive. At their best local settings, CatBoost and LightGBM beat NIAQUE-Global on SMAPE by roughly 0.5–0.6 points and on RMSE by 0.026–0.035, while NIAQUE keeps the best CRPS. Since the abstract and Section 3.2 frame the result as significantly outperforming tree methods, this baseline choice is the central load-bearing weakness. The other issues — missing error bars in Table 2 and impossible confidence intervals in Appendix D.1 — are secondary. The theorem proof is standard and internally consistent. My recommendation is unchanged from the reader's REJECT: the strongest 'best performance' claim is unsupported as written, but a revision that uses fair baselines, softens the superiority claim to calibrated competitiveness, adds seeded variance, and releases code could change the verdict.","tokens_in":30225,"tokens_out":4401,"duration_ms":48880,"concrete_test":"Re-run the Table 1 benchmark using the best local-tree configurations from the paper's own appendices: XGBoost max_depth=8, CatBoost depth=7, LightGBM num_leaves=50, with the same data splits and metrics. If NIAQUE-Global no longer leads on SMAPE/RMSE, the 'significantly outperforming' claim should be replaced by 'competitive, with stronger CRPS/calibration.' Report per-seed confidence intervals for these configurations as well.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 claims NIAQUE-Global 'achieves the best performance, significantly outperforming both traditional tree-based methods and the Transformer baseline.' But the Table 1 tree rows use the weakest settings from the paper's own ablations. Appendix F (Table 10) reports XGBoost-Local at depth 8 with SMAPE 22.4 and RMSE 0.773; Appendix G (Table 11) reports CatBoost-Local at depth 7 with SMAPE 21.5, RMSE 0.761, and CRPS 0.272; Appendix H (Table 14) reports LightGBM-Local with 50 leaves at SMAPE 21.6, RMSE 0.752, and CRPS 0.278. Against these, NIAQUE-Global (Table 1: SMAPE 22.1, RMSE 0.787, CRPS 0.261) is worse on both SMAPE and RMSE, and better only on CRPS and calibration coverage. Thus the headline superiority claim depends on reporting depth-3 / leaf-limited baselines instead of the stronger configurations the paper itself documents. This is load-bearing because the strongest claim is exactly this superiority; without it the paper reduces to 'competitive with well-tuned trees, with better probabilistic calibration plus a transfer-learning win.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NIAQUE, a permutation-invariant encoder-decoder for probabilistic regression, and proposes to pretrain it jointly on many tabular regression datasets and fine-tune it on unseen tasks. Section 2.1 derives the any-quantile loss from the CRPS identity and proves (Theorem 1) that minimizing expected quantile loss recovers the inverse conditional CDF. The authors introduce TabRegSet-101, a 101-dataset regression benchmark, and report experiments on full-benchmark training, held-out transfer, and two Kaggle competitions. Claims include positive transfer from pretraining and significant superiority over tree-based models, Transformer baselines, TabPFN, and TabDPT.","tokens_in":30519,"tokens_out":9477,"duration_ms":100283,"significance":"If the claims held, the main contribution would be meaningful: scalable probabilistic transfer for tabular regression is underexplored, the architecture is more efficient than attention-based alternatives, and the paper ships a reusable benchmark and extensive ablations. The theoretical link between quantile loss and posterior CDFs is standard but cleanly presented, and the held-out transfer experiments (Table 2) show a consistent improvement direction. However, the headline empirical claim of significant superiority over tree-based models is not supported by the paper's own ablations. With the strongest settings the authors themselves report in Appendices F--H, XGBoost, CatBoost, and LightGBM all beat NIAQUE-Global on SMAPE and RMSE. The credible contribution after correcting the comparison is competitive point accuracy plus better probabilistic calibration and a transfer-learning gain, not state-of-the-art point prediction.","major_comments":[{"comment":"The claim that NIAQUE-Global 'significantly outperforms both traditional tree-based methods' is contradicted by the paper's own hyperparameter sweeps when the strongest configurations are used. Table 10 reports XGBoost-Local at depth 8 with SMAPE 22.4, RMSE 0.773; Table 11 reports CatBoost-Local at depth 7 with SMAPE 21.5, RMSE 0.761, CRPS 0.272; Table 14 reports LightGBM-Local with 50 leaves at SMAPE 21.6, RMSE 0.752, CRPS 0.278. Against these, NIAQUE-Global (Table 1: SMAPE 22.1, RMSE 0.787, CRPS 0.261) is worse on SMAPE and RMSE and better only on CRPS and calibration. Table 1 instead reports depth-3 or leaf-limited baselines (XGBoost 25.6, CatBoost 24.3, LightGBM 25.7). The headline comparison must be rebuilt from the best validated configurations, or the claim should be narrowed to 'competitive on point metrics with better probabilistic calibration.'","section":"Section 3.2, Table 1 vs Appendices F, G, H"},{"comment":"The central positive-transfer result is presented without uncertainty quantification. Table 2 reports a single comparison per fine-tuning proportion ps, with no standard errors, confidence intervals, or paired significance tests. Appendix D.3 shows that 4-seed confidence intervals can be computed for the main benchmark, so the absence here is a choice. Since only 21 held-out datasets are used, the consistent but unquantified gap between NIAQUE-Pretrain and NIAQUE-Scratch could depend on a small number of datasets. Please add bootstrap or paired intervals, or otherwise quantify the variability.","section":"Section 3.2, Table 2"},{"comment":"The theorem statement is mathematically imprecise. It defines 'F as a probability measure over variable y' and concludes the minimizer of E ρ(y,F^{-1}(q)) is F = P_{y|x}. If F is a single unconditional distribution independent of x, the minimizer is the marginal P_y, not P_{y|x}; the proof only goes through if F is allowed to depend on x, i.e., F = (F_x)_{x∈X} with F_x^{-1}(q) corresponding to fθ(x,q). Please restate the theorem with conditional distributions F_x and clarify that the minimization is pointwise in x. The underlying standard result is correct, but the current statement is not.","section":"Section 2.1 and Appendix B, Theorem 1"},{"comment":"The Kaggle tree-based baseline rows are adopted from public notebooks and forum posts rather than being run under the authors' own controlled protocol (see Table 7 caption: 'Baseline results are adopted from publicly shared notebooks and discussion forums'). These external results may use different hardware, preprocessing, augmentation, or ensembling. The statement that NIAQUE 'outperforms vanilla tree-based models' in Kaggle settings is therefore not a controlled comparison. The authors should either run the tree baselines under the same protocol or explicitly qualify the claim as 'competitive with published public solutions.'","section":"Section 3.2 and Appendix D.2"}],"minor_comments":[{"comment":"Table 3 labels the second NIAQUE row simply 'NIAQUE-Pretrain', while Appendix D.2.1 distinguishes NIAQUE-Pretrain-100 (Abalone excluded) from NIAQUE-Pretrain-full (Abalone included). Add a footnote in Table 3 to make clear which variant is reported.","section":"Table 3 and Appendix D.2.1"},{"comment":"The phrase 'closes this gap' and 'for the first time' are stronger than the evidence warrants; transfer learning for regression has been studied in related forms. Suggest softening.","section":"Abstract and Section 1"},{"comment":"The 'implicit task ID' hypothesis is presented as the mechanism for transfer, but the supporting evidence is only the qualitative UMAP visualization in Fig. 2. A quantitative analysis (e.g., embedding similarity vs task similarity) would strengthen the claim.","section":"Section 2.4"},{"comment":"Confidence intervals are reported only for a subset of models in Table 9; the main Table 1 lacks them for the full set. Please include CIs for all rows or state why some are omitted.","section":"Section 3.1 and Appendix D.3"},{"comment":"The LightGBM max_depth=2 rows show identical metrics across all leaf counts, which suggests the depth cap dominates and the leaf parameter is inactive. This should be noted when interpreting those baseline settings.","section":"Appendix H, Table 14"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the fairness of the tree-baseline comparison. The authors already have the stronger settings in their own appendices, so this is fixable within the scope of a revision, but the current Table 1 does not support the 'significantly outperforming' claim. I would want to see Table 1 rebuilt with the best validated configurations and the significance language softened before considering acceptance. The transfer claim in Table 2 also needs error bars. If the authors decline to rerun the comparison, rejection would be appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about arXiv:2508.16355 is that its best result is not the one in the headline. The transfer learning part holds up: pretraining a single quantile-conditioned network on 80 datasets and fine-tuning on held-out ones beats training from scratch, especially at low data fractions (Table 2), and the Kaggle experiments include an honest ablation that pretrains without the original Abalone data (Pretrain-100) and still shows a gain. TabRegSet-101 is a useful contribution. That part is worth engaging with.\n\nThe problem is Section 3.2's claim that NIAQUE 'significantly outperforms' tree-based methods. Table 1 reports XGBoost, CatBoost, and LightGBM at their weakest settings. The paper's own appendices show those same families tuned deeper—XGBoost depth 8, CatBoost depth 7, LightGBM 50 leaves—reach SMAPE 21.5–22.4 and RMSE 0.752–0.773 against NIAQUE-Global's 22.1 and 0.787. NIAQUE still wins on CRPS and coverage, which is a real advantage, but the point-accuracy claim reduces to 'competitive with well-tuned trees,' not superiority. That's a load-bearing overstatement.\n\nOther soft spots are minor but real: no code or data release, Table 2 has no error bars, and some confidence intervals in Appendix D.1 are impossible (e.g., XGBoost-Global RMSE 0.075 ± 0.143). Also, the posterior-approximation argument imports Farrell et al. without checking the regularity conditions; the theory itself is just the standard CRPS propriety identity, cleanly re-derived but not new.\n\nOverall this is a competent paper with a useful benchmark, an honest transfer evaluation, and a fixable but significant baselines problem. It deserves a serious referee. The revision should re-run trees at their best settings, add seeded variance, fix the CI machinery, and drop the word 'significantly.' I'd read a revised version again.","headline":"Transfer result is real, but the 'significantly outperforms trees' claim collapses against the paper's own appendix tables; worth reviewing, not desk-rejecting.","tokens_in":31112,"tokens_out":2531,"would_cite":true,"duration_ms":27917,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that probabilistic regression can be solved by one quantile-conditioned network pretrained on many tabular datasets, with positive transfer to new tasks and the largest gains under data scarcity.","keywords":["NIAQUE","probabilistic regression","quantile regression","transfer learning","tabular data","permutation invariance","CRPS","feature importance"],"falsifier":"Run XGBoost-Local at depth 8, CatBoost-Local at depth 7, and LightGBM-Local with 50 leaves on the same 101-dataset protocol and compare SMAPE, RMSE, and CRPS to NIAQUE-Global; if the tuned trees match or beat it, the 'significantly outperforms' claim fails.","tokens_in":30024,"feed_emoji":"📈","tokens_out":11497,"duration_ms":110255,"temperature":0.7,"pith_summary":"The paper introduces NIAQUE, a neural network that takes a tabular data row plus a requested quantile level and outputs that quantile of the target. Its central claim is that one such network can be pretrained jointly on many unrelated regression datasets and then fine-tuned on a new dataset, with positive transfer that is largest when labeled data is scarce. On a new 101-dataset benchmark, the globally pretrained model reports the best aggregate scores, ahead of local and global tree ensembles, a Transformer, TabPFN, and TabDPT. The paper also proves that the expected quantile loss is minimized only when the model outputs the inverse conditional cumulative distribution of the target, which is what makes the predicted quantiles meaningful. If these claims hold, tabular probabilistic regression stops being a per-dataset modeling chore and becomes a transfer-learning problem.","feed_headline":"Quantile net pretrained on 101 sets transfers to new regressions","feed_subtitle":"One any-quantile model fine-tunes to unseen tasks with little data and nearly matches top competition scores.","key_machinery":"The load-bearing mechanism is any-quantile learning: during training each example is paired with q drawn uniformly from (0,1) and scored with the pinball/quantile loss, and Theorem 1 ties this objective to CRPS and to the inverse CDF of the posterior. The architecture that makes transfer work is a prototype-aggregation encoder with dual residual loops: feature values and learned feature-ID embeddings are transformed per-feature, averaged into a fixed-size prototype, and residual blocks refine it while a delta constraint forces each block to encode only deviation from the current prototype. A FiLM-conditioned decoder turns the prototype plus any quantile vector into predictions. This separate","core_discovery":"NIAQUE's central claim is that probabilistic regression can be treated as a single multi-dataset learning problem. The model encodes each feature as a value plus a learned feature-ID embedding, aggregates the per-feature embeddings by averaging into a fixed-size prototype, and a decoder uses FiLM conditioning on the quantile q to emit quantile estimates. Because input dimension is handled by aggregation, datasets with different numbers of columns share one parameter set; the feature-ID embeddings act as an implicit task identifier. The paper proves Theorem 1: over a uniform q, the expected quantile loss equals a scaled CRPS and is minimized if and only if the predicted quantile function is t","pith_inferences":["The feature-ID embeddings are described as an implicit task identifier; a direct test is to permute feature IDs for a held-out dataset before fine-tuning and see whether transfer degrades, separating learned task identity from feature semantics.","The marginal-confidence-interval importance construction only needs quantile outputs, so it could be applied to other probabilistic regressors, not just NIAQUE.","The benchmark samples all datasets uniformly; a curriculum or domain-weighted pretraining mix that emphasizes datasets similar to the target could amplify the reported transfer gains.","Because the model predicts arbitrary quantiles, inference-time quantile selection can adapt it to asymmetric costs or risk-averse decisions without retraining."],"forward_implications":["A single pretrained model can be a reusable starting point for new regression tasks, so small-data practitioners no longer need to fit a model from scratch.","Full quantile curves support calibrated intervals, asymmetric error measures, and decision rules that need more than the median.","The linear O(d+Q) scaling per row makes the architecture practical for datasets with millions of rows, where attention-based tabular models hit memory limits.","Model-native feature importance is available at a fraction of the compute of post-hoc attribution, with high ranking agreement reported.","If transfer generalizes as reported, default practice for tabular regression could move from per-dataset tree tuning to pretrained neural regression plus light fine-tuning."],"supporting_citations":[{"why":"Gives the identity linking CRPS to an integral of quantile loss, which Theorem 1 uses to show the training objective is minimized by the inverse posterior CDF.","marker":"[Gneiting and Ranjan, 2011]"},{"why":"Supplies the large-sample convergence result that lets the paper identify the trained network's output with the inverse conditional CDF.","marker":"[Farrell et al., 2021, Theorem 1]"},{"why":"Provides the any-quantile probabilistic loss and estimation idea that NIAQUE adapts from time-series to tabular regression.","marker":"Smyl et al. [2024]"},{"why":"Contributes the prototype-residual, permutation-invariant encoder that lets one model consume datasets with different feature sets.","marker":"Oreshkin et al. [2022]"},{"why":"Defines the FiLM conditioning mechanism the quantile decoder uses to modulate the observation embedding by q.","marker":"Perez et al. [2018]"},{"why":"TabPFN is the synthetic-pretraining tabular transformer baseline NIAQUE compares against on real competitions.","marker":"[Hollmann et al., 2023]"},{"why":"TabDPT is the real-data-pretraining tabular transformer baseline NIAQUE must beat in the transfer experiments.","marker":"[Ma et al., 2024]"},{"why":"XGBoost is one of the three tree baselines whose local and global versions anchor the benchmark comparison.","marker":"[Chen and Guestrin, 2016]"},{"why":"LightGBM is the second tree baseline; its strongly tuned local variants are the closest tree competitors in the appendix sweeps.","marker":"Ke et al. [2017]"},{"why":"CatBoost is the third tree baseline and the one whose depth-7 local sweep nearly matches NIAQUE in the paper's own ablation tables.","marker":"Prokhorenkova et al. [2019]"}],"fun_headline_variants":["Pretrain once, quantile everywhere: NIAQUE transfers across datasets","Any-quantile net pretrained on many tasks beats tabular baselines","NIAQUE: One model, many regressions, transferable quantiles","Probabilistic transfer learning: Pretrain on 101 sets, fine-tune to new","Neural quantile regression improves via multi-dataset pretraining"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The claim that NIAQUE significantly outperforms tree-based methods rests on the tree hyperparameters used in the main comparison table; the paper's own appendix reports deeper XGBoost, CatBoost, and LightGBM models with aggregate scores close to NIAQUE's, so a fairer baseline choice could shrink the reported margin.","fun_headline_variants_meta":{"raw":{"variants":["Pretrain once, quantile everywhere: NIAQUE transfers across datasets","Any-quantile net pretrained on many tasks beats tabular baselines","NIAQUE: One model, many regressions, transferable quantiles","Probabilistic transfer learning: Pretrain on 101 sets, fine-tune to new","Neural quantile regression improves via multi-dataset pretraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000477,"raw_usage":{"total_tokens":2148,"prompt_tokens":636,"completion_tokens":1512,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":380,"completion_tokens_details":{"reasoning_tokens":1412}},"tokens_in":380,"tokens_out":1512,"duration_ms":10414,"temperature":1.0,"reasoning_tokens":1412,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:24:14.694138+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run XGBoost-Local at depth 8, CatBoost-Local at depth 7, and LightGBM-Local with 50 leaves on the same 101-dataset protocol and compare SMAPE, RMSE, and CRPS to NIAQUE-Global; if the tuned trees match or beat it, the 'significantly outperforms' claim fails.","supporting_citations":[{"cited_title":"Comparing density forecasts using threshold-and quantile-weighted scoring rules","cited_arxiv_id":null,"evidence_quote":"Gives the identity linking CRPS to an integral of quantile loss, which Theorem 1 uses to show the training objective is minimized by the inverse posterior CDF."},{"cited_title":"Deep neural networks for estimation and inference","cited_arxiv_id":null,"evidence_quote":"Supplies the large-sample convergence result that lets the paper identify the trained network's output with the inverse conditional CDF."},{"cited_title":"Film: Visual reasoning with a general conditioning layer","cited_arxiv_id":null,"evidence_quote":"Defines the FiLM conditioning mechanism the quantile decoder uses to modulate the observation embedding by q."},{"cited_title":"Tab PFN : A transformer that solves small tabular classification problems in a second","cited_arxiv_id":null,"evidence_quote":"TabPFN is the synthetic-pretraining tabular transformer baseline NIAQUE compares against on real competitions."},{"cited_title":"Catboost: unbiased boosting with categorical features, 2019","cited_arxiv_id":null,"evidence_quote":"CatBoost is the third tree baseline and the one whose depth-7 local sweep nearly matches NIAQUE in the paper's own ablation tables."}],"review_version":1}