{"id":"ec78806a-95a9-4c3f-b54e-04d9179ba2e6","arxiv_id":"2509.04430","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Modern tabular deep learning methods, including numerical embeddings, retrieval heads, and TabM ensembles, gain most of their advantage on high-data-uncertainty samples, and a triplet-trained embedding built from this insight outperforms standard embeddings.","lead":"This paper argues that modern tabular deep learning methods succeed largely because they handle noisy, ambiguous data points better, and it uses this perspective to build a slightly better numerical feature embedding. It is worth reading because it offers one common explanation for several seemingly unrelated design choices in tabular machine learning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The real-data evidence for \"disproportionate\" benefit on high-uncertainty points rests on absolute ΔMSE curves, which can grow simply because the baseline MSE grows with uncertainty; the paper does not provide a relative-improvement analysis to rule this out.","rationale":"The reader's conditional verdict is appropriate, but their identified weakest assumption (validity of CatBoost uncertainty estimates) is not the most load-bearing issue. The paper actually provides several pieces of evidence that the estimates are meaningful: correlation with an MLP estimator (Figure 2a), validation on synthetic data with known uncertainty (Figure 2b), and consistent conclusions across estimators in Appendix H. Even if the uncertainty estimates are perfect, however, the real-data plots do not establish 'disproportionate' benefit unless the analysis controls for the fact that absolute MSE grows with uncertainty. A constant relative improvement produces exactly the observed qualitative pattern of increasing absolute ΔMSE, so the visual evidence is consistent with a much weaker claim. This concern is concrete and testable: recomputing relative improvements per uncertainty bin would settle it. The practical contribution of the paper, the LRLRtriplet embedding (Table 2), does not depend on this causal attribution and appears to be a genuine, reproducible improvement; the gradient-based analysis of TabM (Figures 11 and 12) also provides independent mechanistic support in a controlled setting. Nevertheless, the broad claim that 'a significant portion of these techniques' success' is due to uncertainty management should not be accepted as established on the current real-data evidence. The reader reached CONDITIONAL, and this critique reinforces rather than changes that verdict; no verdict adjustment is needed.","tokens_in":17643,"tokens_out":6965,"duration_ms":70442,"concrete_test":"For each dataset and model in Figures 6, 8, and 10, bin test points by deciles of estimated uncertainty and compute the per-bin relative improvement (MSE_MLP − MSE_model)/MSE_MLP with bootstrap confidence intervals; test for a monotonic increasing trend across bins (e.g., Spearman correlation or Jonckheere-Terpstra). Additionally, fit ΔMSE as a function of MSE_MLP and check whether the slope significantly exceeds 1 or the intercept is positive, which would indicate super-proportional improvement. If the relative improvement is flat across uncertainty bins, the \"disproportionate\" claim fails and the paper's central attribution reduces to \"better models help most where errors are largest.\" Repeat the same analysis on the synthetic dataset with known true uncertainty values to confirm the result is not an artifact of the estimator.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that numerical embeddings, ModernNCA, and TabM are disproportionately beneficial on high-uncertainty points is supported mainly by plots of absolute ΔMSE = MSE_MLP − MSE_model versus test points sorted by estimated uncertainty (Figures 6, 8, 10). Under the paper's own model y_i = f(x_i) + e^{g(x_i)}·N(0,1), the expected squared error of any model is σ²(x) + excess(x), where σ²(x) = e^{2g(x)} is the irreducible aleatoric variance. The irreducible term cancels when subtracting two models' expected MSE, so an increasing absolute ΔMSE means the excess error (bias plus model-induced variance) is larger on high-σ points. That can reflect a genuinely uncertainty-specific mechanism, but it can also reflect the trivial fact that high-σ points are harder, so any better model gains more absolute error there. In particular, a model that reduces error by a constant relative factor everywhere produces ΔMSE ∝ MSE_MLP, which grows with uncertainty because MSE_MLP itself grows. The paper explicitly claims the growth is faster than MSE_MLP (Section 5) but provides no numerical slopes, normalized curves, or confidence intervals; the visual claim is not quantitatively established. Even on the synthetic dataset with true uncertainty (Figure 3), the same absolute ΔMSE metric is used, so the same artifact applies. Without a relative-improvement analysis, the headline \"disproportionately more beneficial\" and the attribution to uncertainty-management mechanisms are not supported by the real-data evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes data (aleatoric) uncertainty as a unifying lens for explaining why several recent tabular deep learning techniques outperform a plain MLP: numerical feature embeddings (PLR/LRLR), retrieval-augmented models (ModernNCA), and parameter-efficient ensembling (TabM). The authors estimate the conditional target variance using a CatBoost model with an uncertainty-aware loss, sort test examples by estimated uncertainty, and present smoothed curves of absolute MSE differences (MSE_MLP minus MSE_model) to argue that the gains are concentrated on high-uncertainty points. They support this with synthetic datasets with known ground-truth noise, including an MLP-generated heteroscedastic regression, a saw-like 2D regression, and an 'Uncertain CIFAR-10' classification experiment. They also propose a new embedding scheme, LRLR_triplet, trained with a triplet loss to maximize local target consistency, and report improved average rank over standard LRLR embeddings on regression benchmarks.","tokens_in":18000,"tokens_out":12200,"duration_ms":110472,"significance":"If the uncertainty-attribution claim is established, the paper would provide a genuinely unifying explanation for several seemingly unrelated design choices in tabular deep learning and a principled basis for future method development. The paper is strengthened by multi-dataset evidence, synthetic ground-truth validation, robustness checks across alternative uncertainty estimators (Appendix H), and the release of code. The proposed LRLR_triplet embedding is a useful practical artifact. However, the central quantitative support currently rests on visual inspection of absolute-error curves, and the real-data conclusions depend on an uncertainty estimator whose accuracy on real data is not directly validated; both issues are addressable in revision.","major_comments":[{"comment":"The claim that the methods are 'disproportionately more beneficial' on high-uncertainty points is supported only by absolute ΔMSE = MSE_MLP − MSE_model versus estimated uncertainty. Under the paper's own generative model (Eq. 1), the expected MSE of any model is e^{2g(x)} plus an excess-error term, and the irreducible variance cancels in ΔMSE. Thus an increasing absolute ΔMSE is expected for any better model on harder (high-uncertainty) points, even if the model provides a constant relative improvement everywhere; in that case ΔMSE ∝ MSE_MLP, which grows with uncertainty because MSE_MLP itself grows. The paper explicitly claims the growth is faster than MSE_MLP (Section 5), but no slopes, normalized curves, or confidence intervals are reported. The same absolute metric is used in Figure 3 even though true uncertainty is known, so the synthetic validation does not resolve this artifact. Please provide a relative-improvement analysis, e.g., ΔMSE/MSE_MLP as a function of uncertainty or quantitative slope comparisons with confidence intervals, for Figures 3, 6, 8, and 10; without this, the attribution to an uncertainty-specific mechanism is not established.","section":"Sections 5–7 (Figures 3, 6, 8, 10)"},{"comment":"The real-data conclusions require that the CatBoost-based uncertainty estimates order test examples by true aleatoric variance. The only validation against ground truth is on synthetic data with MLP-generated f and g, and Figure 2b shows only a scatterplot without a quantitative accuracy measure. Appendix H demonstrates robustness to the choice of estimator (CatBoost vs MLP/MLP-PLR), but it does not validate any estimator against true uncertainty on the real datasets where the main conclusions are drawn. A concrete test would be to inject controlled label noise into real features (or use datasets with repeated labels) and verify that the estimated-uncertainty ranking correlates with the known noise levels; at minimum, report a rank correlation or calibration statistic for the estimators on real data. Without this, the observed concentration of gains could reflect the estimator's error model rather than true data uncertainty.","section":"Section 3.1, Figure 2b, Appendix H"},{"comment":"The saw-like synthetic dataset is described inconsistently between the main text and the appendix. Section 4.2 samples uniformly from a rectangle with vertices (0,0) and (1,10) and sets the noise standard deviation to e^{g(x)} = x_2^6/62500, depending on the vertical coordinate. Appendix D instead describes triangles with vertices (2i,0), (2i+1,1), (2i+2,0) for i=0..4 (which extend to x_1=10) and a noise standard deviation of x_1^6/4 for x_1 from 0 to 10. The coordinate ranges and the noise formula disagree, so the synthetic experiments in Figures 5, 11, and 12 cannot be reproduced from the paper as written. Please correct the text to match the released code and align the two descriptions.","section":"Section 4.2 and Appendix D"},{"comment":"The comparison between TabM and Deep Ensemble is not controlled for the number of ensemble members. Section 4.2 specifies that the Deep Ensemble averages five independently trained MLPs, while Appendix E fixes 16 branches for TabM; if the same settings are used for Figure 10, the larger effective ensemble size for TabM could be partly responsible for its larger gains on high-uncertainty points. Please compare TabM against a Deep Ensemble with a matched number of members (or otherwise match capacity and computation) before attributing the difference to TabM's parameter-sharing gradient-averaging mechanism.","section":"Section 7, Figure 10, Appendix E"}],"minor_comments":[{"comment":"Please state the Gaussian smoothing bandwidth used for the uncertainty plots and provide a sensitivity check; the smoothing parameter is a free choice that can affect the visual conclusions.","section":"Appendix B"},{"comment":"Please report a rank correlation coefficient (e.g., Spearman) and, if possible, a calibration measure between the CatBoost uncertainty estimates and the ground-truth values; the scatterplot alone does not quantify the claimed strong agreement.","section":"Figure 2b"},{"comment":"The statement that LRLR_triplet 'substantially outperforms' standard LRLR should be supported by the fraction of datasets with statistically significant improvements and by effect sizes, because many dataset-wise differences in Appendix G are within one standard deviation of each other.","section":"Section 5.2, Table 2, Appendix G"},{"comment":"The caption uses 'significantly worse performance' in an informal sense; please either use a statistical test and report the result, or rephrase to avoid implying formal significance testing.","section":"Figure 1 caption"}],"recommendation":"major_revision","confidential_remarks":"This is an interesting analysis paper with a plausible unifying hypothesis, but the central causal claim currently rests on visual inspection of absolute-error curves and on uncertainty estimates whose real-data accuracy is not directly validated. Both issues are addressable in revision: a relative-improvement analysis and a real-data validation of the estimator would substantially raise confidence. The synthetic-data inconsistency should also be fixed. I recommend major revision rather than rejection because the multi-dataset evidence, synthetic ground-truth experiments, estimator-robustness checks, and the proposed embedding are valuable contributions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this paper is the first to put aleatoric data uncertainty at the center of tabular DL design choices, and it shows that numerical embeddings, ModernNCA, and TabM each gain more on high-uncertainty points. Second, the evidence for 'disproportionately more' is thinner than the abstract suggests: it rests on absolute ΔMSE curves sorted by estimated uncertainty, and the paper never quantifies how much faster the gain grows relative to the baseline error.\n\nWhat the paper does well: the synthetic datasets with known noise are used seriously, not as decoration. The TabM gradient decomposition (Figures 11-12) is a concrete, quantitative argument that branch averaging suppresses the noisy gradient component, built from exact synthetic noise. The local-target-consistency analysis for embeddings and the train-set underfitting evidence for ModernNCA provide mechanistic support that goes beyond the uncertainty plots. The proposed LRLRtriplet embedding is a modest but real practical improvement, with per-dataset numbers in the appendix rather than only aggregate ranks.\n\nThe soft spots, in proportion. The stress-test concern is valid: under the paper's own model y = f(x) + e^{g(x)}·N(0,1), any model that improves uniformly in relative terms will show ΔMSE growing with uncertainty, because the baseline MSE itself grows. The paper claims the growth is faster than MSE_MLP but provides no slopes, normalized curves, or confidence intervals. On the synthetic data with true uncertainty, they could compute the relative improvement directly; they don't. That is the main gap. The CatBoost-estimator circularity is a fair worry, but Appendix H shows the same patterns with MLP and MLP-PLR estimators, and Figure 2b shows CatBoost tracks true uncertainty on synthetic data. That concern is minor. Reproducibility is also minor: the code is mentioned as supplementary material but no link is given in the text.\n\nWho gets value: anyone working on tabular DL methods, especially embeddings or ensembling. The overarching explanation is plausible, and the mechanism sections are worth reading even if the headline claim needs qualification. I would send this out for review. The main requested revision should be a quantitative relative-improvement or excess-error decomposition on the synthetic data, so the 'disproportionate' claim is either supported or softened.","headline":"A useful uncertainty lens for tabular DL with solid mechanism sections, but the headline 'disproportionate benefit' claim needs quantitative backup beyond absolute ΔMSE curves.","tokens_in":18506,"tokens_out":3367,"would_cite":true,"duration_ms":30824,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A large share of tabular deep learning's recent wins comes from implicitly handling high data uncertainty, and making that explicit yields a better embedding.","keywords":["data uncertainty","aleatoric uncertainty","tabular deep learning","numerical feature embeddings","retrieval-augmented models","parameter-efficient ensembling","local target consistency","triplet loss"],"falsifier":"On a real regression dataset with clean targets, inject known heteroscedastic noise; if the performance gains of MLP-PLR, ModernNCA, and TabM do not concentrate on samples with the largest injected noise, the paper's main claim fails.","tokens_in":17460,"feed_emoji":"🎯","tokens_out":6626,"duration_ms":51098,"temperature":0.7,"pith_summary":"The paper argues that data uncertainty, meaning the irreducible aleatoric noise in target labels, is the key to understanding why several recent tabular deep learning techniques succeed. Concretely, it claims that numerical feature embeddings, retrieval-augmented models such as ModernNCA, and parameter-efficient ensembling such as TabM all produce their largest gains precisely on test points with the highest data uncertainty. Each technique manages this uncertainty through its own mechanism: embeddings make local neighborhoods more target-consistent, retrieval averages over many neighbors, and TabM averages gradients across branches to damp the noise component. The authors then turn this insight into a practical result: a new embedding scheme, LRLRtriplet, trained with a triplet loss to explicitly maximize local target consistency, which beats standard embeddings on regression benchmarks.","feed_headline":"Tabular deep learning wins by managing uncertain data points","feed_subtitle":"Embeddings, retrieval, and ensembling all gain by taming label noise, leading to a better embedding.","key_machinery":"The central object is the per-sample data uncertainty estimate, obtained by fitting the model $y_i = f(x_i) + e^{g(x_i)} \\cdot \\mathcal{N}(0,1)$ to predict both the noiseless target and the noise scale, using CatBoost with a Gaussian likelihood loss. Around this, the paper builds uncertainty plots that sort test points by estimated uncertainty and plot per-sample MSE differences between models. For the new embedding method, the load-bearing device is the triplet loss: for each anchor, the closer-in-target object is the positive and the farther one the negative, so the embedder is trained to make embeddings reflect target similarity; this pretrained embedder's last linear layer is discarded and the rest initializes the LRLR embedding of an MLP. For TabM, the key identity is the gradient decomposition $\\frac{\\partial(\\phi(x_i)-y_i)^2}{\\partial\\theta} = 2(\\phi(x_i)-f(x_i))\\frac{\\partial\\phi(x_i)}{\\partial\\theta} - 2 e^{g(x_i)}\\mathcal{N}(0,1)\\frac{\\partial\\phi(x_i)}{\\partial\\theta}$, which lets the paper show in synthetic experiments that sharing parameters and averaging gradients across branches suppresses the noisy term more than the clean term.","core_discovery":"The central claim is that a significant portion of the success of numerical feature embeddings, retrieval-augmented models, and advanced ensembling in tabular deep learning can be attributed to their implicit yet effective mechanisms for handling datapoints with high data uncertainty. The paper demonstrates, on real and synthetic regression datasets, that the per-sample performance gap between a plain MLP and each of these techniques grows as estimated uncertainty grows, often faster than the MLP's own error. For embeddings, the mechanism is improved local target consistency: the learned feature space makes neighbors more likely to share similar targets, which matters most where labels are noisy. For retrieval (ModernNCA), averaging over many neighbors resists memorizing label noise, though it can underfit in low-uncertainty regions. For ensembling (TabM), averaging gradients across shared branches reduces the noisy component of the gradient relative to the clean component, making training robust at high uncertainty. As a direct payoff, the paper introduces LRLRtriplet embeddings, which are pretrained with a triplet loss to maximize local target consistency and then fine-tuned; this scheme achieves an average rank of 1.17 across regression datasets compared to 1.67 for standard LRLR embeddings and 2.75 for a plain MLP.","pith_inferences":["Beyond the paper: if the shared mechanism is real, then the unresolved DL-versus-GBDT gap on tabular data is largely a high-uncertainty phenomenon; combining uncertainty-aware losses with GBDT-style smoothing could close it further.","Beyond the paper: the triplet-pretraining recipe could be adapted to classification by treating targets as soft labels, or to tabular foundation models to make their few-shot gains interpretable through uncertainty handling.","Beyond the paper: the analysis only covers regression; a direct test would inject heteroscedastic noise into clean classification datasets and check whether the same techniques' gains concentrate on noisy points."],"forward_implications":["Numerical feature embeddings, retrieval heads, and parameter-efficient ensembles share a common explanatory mechanism; future tabular designs can be evaluated by whether they improve handling of high-uncertainty samples.","The LRLRtriplet scheme offers a concrete accuracy gain on regression benchmarks, with an average rank of 1.17 versus 1.67 for standard LRLR embeddings.","Uncertainty plots become a diagnostic tool: comparing per-sample errors sorted by estimated uncertainty reveals why one method beats another on a given dataset.","TabM's gradient-averaging analysis predicts that ensembling variants that reduce the noisy gradient component will be especially robust to label noise."],"supporting_citations":[{"why":"Introduces the LRLR/PLR numerical feature embeddings whose success is explained and which the new scheme extends.","marker":"Gorishniy et al. (2022)"},{"why":"Introduces ModernNCA, the retrieval-augmented model analyzed in Section 6.","marker":"Ye et al. (2024)"},{"why":"Introduces TabM, the parameter-efficient ensembling model analyzed in Section 7.","marker":"Gorishniy et al. (2025)"},{"why":"CatBoost is the estimator used to produce per-sample data uncertainty values.","marker":"Prokhorenkova et al. (2018)"},{"why":"Supplies the definition of data (aleatoric) uncertainty that frames the whole analysis.","marker":"Gal et al. (2016)"},{"why":"Provides the likelihood-based heteroscedastic regression approach the paper aligns with for uncertainty estimation.","marker":"Duan et al. (2020)"},{"why":"Defines the triplet loss used to pretrain the new LRLRtriplet embeddings.","marker":"Schroff et al. (2015)"},{"why":"Provides the regression benchmark datasets used to evaluate the new embedding scheme.","marker":"Grinsztajn et al. (2022)"},{"why":"Supplies the large-scale, temporally shifted benchmarks (Maps Routing, Sberbank Housing, etc.) used in the uncertainty plots.","marker":"Rubachev et al. (2025)"}],"fun_headline_variants":["Why tabular deep learning works: taming data uncertainty","Data uncertainty hides in plain sight for tabular deep learning","Tabular deep learning's winning move: handling uncertain data","Uncertain data points drive tabular deep learning success"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The CatBoost-based estimates of per-sample data uncertainty reflect the true aleatoric uncertainty on the real datasets; if that mapping is wrong, the apparent concentration of performance gains on 'high-uncertainty' points could be an artifact of the estimator rather than a property of the data.","fun_headline_variants_meta":{"raw":{"variants":["Why tabular deep learning works: taming data uncertainty","Data uncertainty hides in plain sight for tabular deep learning","Tabular deep learning's winning move: handling uncertain data","Uncertain data points drive tabular deep learning success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1463,"prompt_tokens":965,"completion_tokens":498,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":432}},"tokens_in":581,"tokens_out":498,"duration_ms":4840,"temperature":1.0,"reasoning_tokens":432,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:28:57.201955+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a real regression dataset with clean targets, inject known heteroscedastic noise; if the performance gains of MLP-PLR, ModernNCA, and TabM do not concentrate on samples with the largest injected noise, the paper's main claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the definition of data (aleatoric) uncertainty that frames the whole analysis."},{"cited_title":"Y., Thai, K","cited_arxiv_id":null,"evidence_quote":"Provides the likelihood-based heteroscedastic regression approach the paper aligns with for uncertainty estimation."},{"cited_title":"Facenet: A unified embedding for face recognition and clustering","cited_arxiv_id":null,"evidence_quote":"Defines the triplet loss used to pretrain the new LRLRtriplet embeddings."},{"cited_title":"Datasets and Benchmarks","cited_arxiv_id":null,"evidence_quote":"Provides the regression benchmark datasets used to evaluate the new embedding scheme."},{"cited_title":"TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks","cited_arxiv_id":null,"evidence_quote":"Supplies the large-scale, temporally shifted benchmarks (Maps Routing, Sberbank Housing, etc.) used in the uncertainty plots."}],"review_version":2}