REVIEW 3 major objections 3 minor 1 cited by
LakeMLB: Data Lake Machine Learning Benchmark
T0 review · 3 major / 3 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read LakeMLB claims to be the first benchmark for multi-table machine learning in data lakes, reporting that pre-training excels on union tables and feature augmentation on join tables.
desk verdict Useful benchmark resource whose headline FA-in-Join result may be a label-leakage artifact. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing component is the benchmark design itself: each instance is a target table paired with one auxiliary table, with join alignments constructed by embedding entity-name columns with BERT-base-uncased and retrieving the cosine-similarity 1-nearest neighbor. That weak entity matching converts joinable tables into horizontally enriched feature tables, and the same 1-NN machinery (over row-text embeddings) is used to attach auxiliary features in union settings. The three integration strategies—pre-training (PT), data augmentation (DA), and feature augmentation (FA)—are defined on top of this pairing, and fixed train/validation/test splits plus balanced labels make results comparable
What would settle it
Manually label a random sample of the 1-NN entity matches in NNStocks or DSMusic and measure matching precision; or rerun the feature augmentation experiments using ground-truth join keys instead of the fuzzy 1-NN alignments. If matching precision is low, or if FA with true keys does not outperform FA with the benchmark's alignments, the paper's central claim about feature augmentation in joins is not supported.
Extended reading notes
Core claim
LakeMLB is a benchmark suite built around pairs of target and auxiliary tables, one pair per scenario, with three datasets for Union (traffic collisions, building violations, used cars) and three for Join (stock sectors twice, music genres). The paper's claim is that this is the first standardized benchmark for the model-learning stage in data lakes. Across experiments with tree ensembles, tabular neural networks, transfer-learning models, and tabular foundation models, the paper finds that pre-training on the auxiliary table yields the most consistent gains in Union scenarios (83.3% win rate), feature augmentation via 1-NN concatenation yields the largest gains in Join scenarios (91.7% win
Load-bearing premise
The join scenarios (and the FA/DA strategies that exploit them) depend on the assumption that BERT-base-uncased embeddings plus cosine 1-NN retrieval create row/entity alignments that are good enough to be useful; if the alignments are wrong, the join tasks are dominated by matching noise and the reported gains would be artifacts of benchmark construction rather than properties of multi-table learning.
Editorial extensions
If this is right
- Pre-training on a unionable auxiliary table is a dependable way to boost target accuracy, while data augmentation in union settings often hurts because label taxonomies differ across sources.
- Feature augmentation in join settings pays off only when row matching is accurate; deliberately deteriorating the match (farther k-NN or random) degrades accuracy, so entity-resolution quality gates the benefit.
- Transfer-learning tabular models, especially those that exploit text in schemas and cell values, are the strongest overall family in both scenarios, suggesting they are promising for real lake workloads.
- LakeMLB's fixed splits and public code provide a reproducible reference point for comparing future multi-table tabular methods.
- The benchmark's scope is limited to two-table, balanced classification tasks; extending to more tables, regression, and imbalanced labels is the stated next step.
Reading between the lines
- The join-scenario results likely overstate feature augmentation's real-world value, because the fuzzy BERT-based alignments may be tuned to entity-name matching; testing FA with true foreign keys or a manually validated join would isolate how much of the +2.97 pp gain is due to alignment quality rather than the strategy itself.
- A natural extension is to measure the precision of the 1-NN entity matching on a labeled sample; if precision is low, the benchmark's join tasks become matching-noise benchmarks, and FA gains should be re-examined.
- The paper's strategy rankings suggest a practical rule of thumb for data-lake pipelines: when auxiliary tables share a label space, pre-train; when they share entities, augment features; avoid blind row-level data augmentation across sources.
- The benchmark's two-table simplification likely underestimates the value of data augmentation in lakes with many overlapping sources; a version with multiple auxiliary tables could change the DA conclusions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces LakeMLB, a benchmark for multi-table machine learning in data lakes, with two scenarios (Union and Join), six real-world datasets, fixed 70/10/20 splits, and three integration strategies: pre-training (PT), data augmentation (DA), and feature augmentation (FA). The authors evaluate tree-based, deep tabular, transfer-learning, and tabular foundation models, reporting that PT is best in Union scenarios, FA is best in Join scenarios, and transfer-learning models are generally robust. Code and datasets are released. If the construction is sound, LakeMLB fills a real gap between data-discovery benchmarks and single-table tabular benchmarks.
Significance. The benchmark addresses an important gap: most tabular benchmarks are single-table, while data-discovery benchmarks stop at table retrieval. LakeMLB provides a concrete, reproducible evaluation platform for the model-learning stage over unionable and joinable tables, using real-world public data with fixed splits and a unified protocol. The breadth of baselines (tree ensembles, single-table transformers, transfer learners, and foundation models) is a strength, and the release of code and datasets is valuable to the community. However, the empirical conclusions currently rest on two fragile pillars: possible label leakage via auxiliary-table columns in the Join scenarios and the absence of any variance/error-bar reporting for comparisons whose differences are sometimes only one or two percentage points on test sets of a few hundred rows. These issues need to be resolved before the benchmark and its findings can be relied upon.
major comments (3)
- [Appendix A.2 and B.3] The Join auxiliary tables are described as containing raw Wikipedia infobox attributes (NNWiki, LHWiki) and Spotify track metadata (DSMusic), while the target labels are company sector and music genre. For DSMusic, the paper explicitly removes the Discogs 'styles' column to avoid label leakage, but it is silent on whether the Spotify auxiliary table's 'genre' column was removed; the named public Spotify dataset commonly contains such a column. For NNWiki/LHWiki, no analogous removal of infobox fields such as 'Industry' or 'Type' is documented. Under FA, B.3 concatenates the auxiliary attributes into the target rows for both training and test, so if any label-equivalent column remains, the model can observe the target label directly. This would trivially explain the +2.97 pp average FA gain and 91.7% win rate in Table 4. The authors must audit the released auxiliary tables, remove or othe
- [Tables 2–3 and Appendix C.2] Appendix C.2 states that deep baselines are re-trained for 10 runs and that mean and standard deviation are reported. Tables 2 and 3, however, contain bare accuracy numbers with no variance or confidence intervals. This matters for the paper's central ranking: NNStocks and LHStocks have about 216–224 test rows, so a 1–3 pp accuracy difference corresponds to only a handful of instances. The observed FA advantage in Join scenarios is of exactly this size. The authors should report means ± std or confidence intervals across seeds (and, ideally, across split variability), or explicitly state that the numbers are single-run results and temper the conclusions accordingly.
- [Section 5.4.2 / Figure 4] The experiment varies the k-th nearest neighbor (20, 21, 22, 24, and random) and interprets the associated accuracy decline as 'matching quality deteriorates.' But changing k changes the retrieved row, not a measured notion of match quality; the paper never reports whether the 1-NN match is actually correct or how match accuracy degrades with k. The conclusion that FA 'critically relies on high-quality one-to-one row matching' is therefore only weakly supported. Reporting entity-match accuracy on a sample of hand-labeled or exact-match pairs, or comparing against an exact-key join, would make the claim concrete.
minor comments (3)
- [Table 1 vs. Appendix A.1] The GACars auxiliary table is listed as 3,600 rows in Table 1 but as 36,000 samples in Appendix A.1. Please correct this inconsistency; it affects reproducibility and the DA cap of 30%.
- [Figure 4 and Table 3] The x-axis labels '20 21 22 24' in Figure 4 are ambiguous; they should be typeset as 2^0, 2^1, 2^2, 2^4. Also, the caption of Table 3 should indicate whether the reported numbers are means over runs or a single representative run.
- [General documentation] The paper would benefit from a data-card-style appendix listing, for each auxiliary table, the full set of columns and which columns were dropped and why. This is standard practice for benchmarks and would also have made the label-leakage situation self-evident.
Circularity Check
No confirmed circular derivation; empirical benchmark with held-out evaluation.
full rationale
LakeMLB is an empirical benchmark, not a derived result: the central claims (fixed dataset statistics, strategy win rates, and accuracy tables) come from running public tabular models on fixed train/validation/test splits, with hyperparameters selected on validation and final accuracy reported on the target-table test set. No equation fits a parameter to the quantity later presented as a prediction, and no external theorem from the authors' prior work is invoked to force a conclusion. The only overlapping-author citation is [19] (rLLM/SJTUTables), used in the related-work survey; it is not load-bearing for any LakeMLB finding. The BERT+1NN fuzzy matching is used both to construct weak joins and to implement FA, but this symmetry does not by itself force FA's 91.7% win rate: matching does not use target labels, and Section 5.4.2 shows accuracy degrades as the neighbor rank increases or becomes random, indicating sensitivity to matching quality rather than a tautological result. One data-construction risk should be flagged: Appendix A.2 explicitly removes the Discogs 'styles' column 'to avoid potential label leakage' but does not state that label-equivalent auxiliary columns (e.g., Spotify genre or Wikipedia industry) were dropped before FA concatenation; if retained, those fields would leak the target label into FA features. This is a potential leakage/validity flaw, not a confirmed circular derivation, because the paper text does not exhibit the target label entering the features by construction. Therefore no circular step is confirmed; score 0.
Assumptions & free parameters
free parameters (5)
- DA sample-appending cap =
30% of target-table training size
- Auxiliary feature sparsity cutoff =
>95% columns removed
- GACars price bin width =
10,000 currency units
- Fuzzy matching embedding/model =
BERT-base-uncased + cosine 1-NN
- Target table split ratio =
70/10/20 train/val/test
assumptions (5)
- domain assumption BERT-based cosine 1-NN retrieval produces semantically meaningful weak joins between target and auxiliary tables.
- domain assumption Auxiliary tables never enter the test or evaluation stage, so there is no label leakage through auxiliary rows.
- domain assumption The six table pairs are representative of real data lake workloads and the standard unionability/joinability definitions apply.
- domain assumption Public data sources and the described filters yield accurate labels and features for the prediction tasks.
- domain assumption BERT embeddings of category names align label spaces across unionable tables.
Cite this review
Pith. "Pith review of LakeMLB: Data Lake Machine Learning Benchmark." pith.science (2026). https://pith.science/paper/IDCVGEMS
@misc{pith2026260210441,
author = {Pith},
title = {Pith review of: LakeMLB: Data Lake Machine Learning Benchmark},
year = {2026},
howpublished = {\url{https://pith.science/paper/IDCVGEMS}},
note = {Machine review of arXiv:2602.10441}
}
read the original abstract
Data lakes have become a fundamental platform for large-scale machine learning by enabling flexible management of heterogeneous data. Despite their growing importance, standardized benchmarks for evaluating machine learning performance in data lake environments remain scarce. To address this gap, we present LakeMLB (Data Lake Machine Learning Benchmark), the first benchmark designed for multi-table machine learning in data lakes. LakeMLB focuses on two representative scenarios, Union and Join, and provides six real-world datasets spanning diverse domains. It supports three representative multi-table learning paradigms: pre-training, data augmentation, and feature augmentation, together with standardized data splits and evaluation protocols. We conduct extensive experiments with state-of-the-art tabular learning methods and provide insights into their performance across different data lake scenarios. We release both datasets and code to facilitate rigorous research on machine learning in data lake ecosystems; the benchmark is available at https://github.com/zhengwang100/LakeMLB.
Figures
Forward citations
Cited by 1 Pith paper
-
Beyond IID: How General Are Tabular Foundation Models, Really?
Tabular foundation models excel on tiny- to medium-sized IID data but are outperformed by traditional tree-based and deep learning models on non-IID, large, and high-dimensional datasets, based on evaluations across 1...
Reference graph
Works this paper leans on
-
[1]
Michael Armbrust, Ali Ghodsi, Reynold Xin, and Matei Zaharia. 2021. Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics. InAnnual Conference on Innovative Data Systems Research, Vol. 8. 28
2021
-
[2]
Arthur Asuncion, David Newman, et al. 2007. UCI machine learning repository
2007
-
[3]
Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael Gomes Mantovani, Jan N van Rijn, and Joaquin Vanschoren. [n. d.]. OpenML Benchmarking Suites. InThirty-fifth Conference on Neural Infor- mation Processing Systems Datasets and Benchmarks Track
-
[4]
Jintai Chen, Jiahuan Yan, Qiyuan Chen, Danny Z Chen, Jian Wu, and Jimeng Sun
-
[5]
Kuan-Yu Chen, Ping -Han Chiang, Hsin -Rung Chou, Ting -Wei Chen, and Tien-Hao Chang. 2023. Trompt: Towards a Better Deep Neural Network for Tabular Data. InInternational Conference on Machine Learning, Vol. 202. PMLR, 4392–4434
2023
-
[6]
Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A Scalable Tree Boosting System. InACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 785–794
2016
-
[7]
Zi-Jian Cheng, Zi-Yi Jia, Zhi Zhou, Lan-Zhe Guo, and Yu-Feng Li. 2025. TabFS- Bench: Tabular Benchmark for Feature Shifts in Open Environments. InInterna- tional Conference on Machine Learning, Vol. 267. 10025–10089
2025
-
[8]
Yuhao Deng, Chengliang Chai, Lei Cao, Qin Yuan, Siyuan Chen, Yanrui Yu, Zhaoze Sun, Junyi Wang, Jiajun Li, Ziqi Cao, Kaisen Jin, Chi Zhang, Yuqing Jiang, Yuanfang Zhang, Yuping Wang, Ye Yuan, Guoren Wang, and Nan Tang. 2024. LakeBench: A Benchmark for Discovering Joinable and Unionable Tables in Data Lakes.Proceedings of the VLDB Endowment17, 8 (2024), 1925–1938
2024
Show all 32 references
-
[9]
Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmüller, Pra- teek Mutalik Desai, Frank Salinas, and Frank Hutter. 2025. TabArena: A Living Benchmark for Machine Learning on Tabular Data.arXiv preprint arXiv:2506.16791(2025)
2025 arXiv
-
[10]
Josh Gardner, Zoran Popovic, and Ludwig Schmidt. 2023. Benchmarking Distri- bution Shift in Tabular Data with TableShift. InAdvances in Neural Information Processing Systems (Datasets and Benchmarks Track)
2023
-
[11]
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. 2021. Revisiting Deep Learning Models for Tabular Data. InAdvances in Neural Infor- mation Processing Systems, Vol. 34. 18932–18943
2021
-
[12]
Rihan Hai, Sandra Geisler, and Christoph Quix. 2016. Constance: An intelligent data lake system. InACM SIGMOD International Conference on Management of Data. 2097–2100
2016
-
[13]
Alon Halevy, Flip Korn, Natalya F Noy, Christopher Olston, Neoklis Polyzotis, Sudip Roy, and Steven Euijong Whang. 2016. Goods: Organizing google’s datasets. InACM SIGMOD International Conference on Management of Data. 795–806
2016
-
[14]
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. 2023. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second. InInternational Conference on Learning Representations
2023
-
[15]
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. 2020. Tabtrans- former: Tabular data modeling using contextual embeddings.arXiv preprint arXiv:2012.06678(2020)
2020 arXiv
-
[16]
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. InAdvances in Neural Information Processing Systems. 3149–3157
2017
-
[17]
Miller, and Mirek Riedewald
Aamod Khatiwada, Grace Fan, Roee Shraga, Zixuan Chen, Wolfgang Gatterbauer, Renée J. Miller, and Mirek Riedewald. 2023. SANTOS: Relationship-based Seman- tic Table Union Search. InACM SIGMOD International Conference on Management of Data
2023
-
[18]
Myung Jun Kim, Léo Grinsztajn, and Gaël Varoquaux. 2024. CARTE: Pretrain- ing and Transfer for Tabular Learning. InInternational Conference on Machine Learning, Vol. 235. 23843–23866
2024
-
[19]
Weichen Li, Xiaotong Huang, Jianwu Zheng, Zheng Wang, Chaokun Wang, Li Pan, and Jianhua Li. 2024. rLLM: Relational table learning with LLMs.arXiv preprint arXiv:2407.20157(2024)
2024
-
[20]
Fatemeh Nargesian, Erkang Zhu, Ken Q Pu, and Renée J Miller. 2018. Table union search on open data.Proceedings of the VLDB Endowment11, 7 (2018), 813–825
2018
-
[21]
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Doro- gush, and Andrey Gulin. 2018. CatBoost: Unbiased Boosting with Categorical Features. InAdvances in Neural Information Processing Systems. 6638–6648
2018
-
[22]
Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. 2025. TabICL: A Tabular Foundation Model for In-Context Learning on Large Data. In International Conference on Machine Learning, Vol. 267. PMLR, 50817–50847
2025
-
[23]
Joshua Robinson, Rishabh Ranjan, Weihua Hu, Kexin Huang, Jiaqi Han, Alejandro Dobles, Matthias Fey, Jan Eric Lenssen, Yiwen Yuan, Zecheng Zhang, Xinwei He, and Jure Leskovec. 2024. RelBench: A Benchmark for Deep Learning on Relational Databases. InAdvances in Neural Informatio...
2024
-
[24]
Ivan Rubachev, Nikolay Kartashev, Yury Gorishniy, and Artem Babenko. 2025. TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Bench- marks. InInternational Conference on Learning Representations
2025
-
[25]
Pegdwendé Sawadogo and Jérôme Darmont. 2021. On data lake architectures and metadata management.Journal of Intelligent Information Systems56, 1 (2021), 97–120
2021
-
[26]
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. 2021. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training.arXiv preprint arXiv:2106.01342(2021)
2021 arXiv
-
[27]
Kavitha Srinivas, Julian Dolby, Ibrahim Abdelaziz, Oktie Hassanzadeh, Harsha Kokel, Aamod Khatiwada, Tejaswini Pedapati, Subhajit Chaudhury, and Horst Samulowitz. 2023. LakeBench: Benchmarks for Data Discovery over Data Lakes. arXiv preprint arXiv:2307.04217(2023)
2023 arXiv
-
[28]
Minjie Wang, Quan Gan, David Wipf, Zhenkun Cai, Ning Li, Jianheng Tang, Yanlin Zhang, Zizhao Zhang, Zunyao Mao, Yakun Song, Yanbo Wang, Jiahang Li, Han Zhang, Guang Yang, Xiao Qin, Chuan Lei, Muhan Zhang, Weinan Zhang, Christos Faloutsos, and Zheng Zhang. 2024. 4DBInfer: A 4D ...
2024
-
[29]
Zifeng Wang and Jimeng Sun. 2022. TransTab: Learning Transferable Tabu- lar Transformers Across Tables. InAdvances in Neural Information Processing Systems
2022
-
[30]
Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. 2023. XTab: Cross-table Pretraining for Tabular Transformers. InInter- national Conference on Machine Learning, Vol. 202. 43181–43204
2023
-
[31]
Erkang Zhu, Dong Deng, Fatemeh Nargesian, and Renée J Miller. 2019. JOSIE: Overlap Set Similarity Search for Finding Joinable Tables in Data Lakes. InACM SIGMOD International Conference on Management of Data. 847–864. Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Fei...
2019
-
[2024]
InACM SIGKDD Conference on Knowledge Discovery and Data Mining
Can a deep learning model be a sure bet for tabular prediction?. InACM SIGKDD Conference on Knowledge Discovery and Data Mining. 288–296
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.