{"id":"4d814bc4-767e-4461-b067-6478fdf22bb1","arxiv_id":"2507.04687","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LAKEGEN builds synthetic, domain-specific tabular benchmarks using ontologies and an LLM, and shows current dataset discovery methods struggle on the resulting semantic joinability tasks.","lead":"LAKEGEN uses an LLM to generate domain-specific tables and joinability labels from an ontology, producing synthetic data lakes for finance, pharma, and healthcare. The generated benchmarks are harder for current joinability discovery methods than existing corpora, which the authors argue shows those corpora under-test discovery tools.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed benchmark difficulty rests on unvalidated LLM-generated semantic-join ground truth; without a check that perturbed values remain semantically equivalent, the low F1 scores could be label-noise artifacts.","rationale":"The reader's verdict identifies realism of LLM-generated tables and ground truth as the weakest assumption. I agree that this is the crux, but the most load-bearing subset is the correctness of the semantic-join ground truth, because without it the experimental section cannot support any conclusion about method performance. The exact-join labels are grounded in value containment (Section 3.2) and are relatively safe. The semantic-join labels, however, are produced by an LLM perturbation step with no verification; the paper itself notes how easy it is to break semantics. Since Tables 3-5 and the headline 'current datasets fail to evaluate these methods properly' rest on these labels, a label-noise artifact would invalidate the central claim. This is a falsifiable, fixable issue rather than a fundamental flaw; a human-annotation study on a sample of pairs would settle it. I therefore keep the reader's CONDITIONAL verdict, and mark partial agreement because I emphasize internal label validity more than external data realism. The paper is otherwise well-structured, gives a clear description of the pipeline, and shows consistent experimental patterns; the concern is not about novelty or implementation, but about the validity of the self-generated ground truth.","tokens_in":13433,"tokens_out":6354,"duration_ms":70121,"concrete_test":"Sample 100 'semantic join' pairs per domain from the generated corpora, stratified by the 'difficult' vs 'easy' categories of Table 5; have two independent annotators with domain familiarity judge whether the two value sets are semantically joinable (i.e., every target value has a plausible equivalent in the source, allowing standard semantic mappings). Compute agreement between LAKEGEN's ground truth and the annotators, and report per-domain accuracy. If accuracy is below 90% or inter-annotator agreement is low, re-run Tables 3-5 on the human-verified subset; if method rankings or relative gaps change, the original conclusions are not robust to label correctness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that LAKEGEN produces corpora that 'pose multiple challenges' to discovery methods and reveal that current benchmarks under-evaluate them—depends on the correctness of the ground-truth joinability labels. For semantic joins, ground truth is not human-annotated; it is inferred from ontology data types (Section 3.2) and from LLM-based 'semantically-safe value perturbations' (Section 4.1). The paper acknowledges that simple value substitutions can break semantics ('123 Main Street' is not '123 Principal Street'), and it provides only anecdotal evidence that text-davinci-003 avoids this. There is no evaluation of the fraction of generated 'semantic join' pairs that are actually semantically equivalent, nor of whether the 'difficult non-exactly joinable pairs' in Table 5 are truly joinable. If even a minority of these labels are wrong, the lower F1 and top-k scores of COMA, JL, SF, and Starmie reflect label noise rather than genuine failure on semantic joins, and the conclusion that current datasets fail to evaluate methods properly is unsupported. The concern is about internal validity, not just external realism, and it can be settled with a focused human-annotation study.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents LAKEGEN, a pipeline that takes a domain ontology, maps concepts and data properties to table schemas, uses an LLM (text-davinci-003) to populate rows while maintaining inter-table dependencies, and applies table-shape, schema, and column-value perturbations to create derived tables. Joinability ground truth is inferred from ontology object properties and from LLM-based semantic perturbations. The authors generate three corpora (finance, pharma, healthcare) and evaluate COMA, Jaccard-Levenshtein, Similarity Flooding, and Starmie on exact and semantic joinability, reporting low F1 and top-k scores and arguing that existing public corpora are too easy to expose these weaknesses.","tokens_in":13674,"tokens_out":6608,"duration_ms":66503,"significance":"If the semantic-join labels are correct, LAKEGEN would provide a practical way to generate domain-specific, ontology-driven benchmarks with controlled joinability relationships, and the experiments would reveal that current discovery methods, especially the embedding-based Starmie, underperform on semantic joins. The paper's use of MIMIC-III base tables for healthcare is a valuable realism anchor, and the perturbation catalogue in Table 7 is a useful systematization. However, the absence of any human evaluation of the semantic-join ground truth is a substantive gap that currently prevents the central difficulty claim from being accepted: the low scores could be label-noise artifacts rather than evidence of genuine failure on semantic joins. The paper also overstates its contributions with an unsubstantiated 'human-annotated' claim and a comparison only against three simple public benchmarks.","major_comments":[{"comment":"The ground truth for semantically joinable pairs is generated by the same LLM perturbation pipeline that creates the tables, but the paper never validates that the perturbed values remain semantically equivalent to the originals. The '123 Main Street' -> '123 Main St.' example is anecdotal. If a nontrivial fraction of the generated semantic-join pairs are not actually joinable, the low F1 and top-k scores in Tables 3–5 reflect label noise rather than genuine difficulty. This is particularly relevant for the 'difficult non-exactly joinable pairs' in Table 5 (e.g., 27 evaluated for pharma, 70 for healthcare), where no check is reported that the columns are truly semantically joinable. I request a label-quality study: sample about 100 semantic-join pairs per domain, have two annotators judge joinability, report inter-annotator agreement and the fraction of pairs judged non-joinable, and rerun the main tables excluding or re-labeling incorrect pairs.","section":"Section 4.1, Tables 3–5"},{"comment":"The abstract criticizes existing corpora for not containing human-annotated column pairs, and Table 1 marks a 'Human-annotated' checkmark for LAKEGEN, but Section 3.1 states that joinability is inferred from ontology object properties and no human annotation step is described anywhere in the paper. This is a factual overclaim that should be corrected or substantiated with an actual annotation procedure. Relatedly, the introduction claims 'As we show in Section 3.2, LLMs score equally well in generating domain-specific datasets,' but Section 3.2 contains no score comparison—only an anecdotal remark about city/zip matches. These claims should be removed or properly supported.","section":"Abstract, Table 1, Section 3.1"},{"comment":"The conclusion that 'current datasets fail to evaluate these methods properly' is supported only by comparing performance on the generated base tables with Gosales, TPC-H, and TPC-DS. Those are not the corpora criticized in Section 1 (e.g., GitTables, Valentine, Magellan), and no experiments are run on those criticized corpora. Moreover, the abstract and introduction claim that LAKEGEN creates challenges for both 'union search and joinability discovery methods,' but the experimental section evaluates only joinability (exact and semantic). Either add union-search experiments and comparisons on the criticized corpora, or soften the conclusions to the specific settings actually tested.","section":"Section 5.1, Table 6"},{"comment":"The paper repeatedly describes the generated corpora as 'realistic' domain-specific data lakes, but it provides no validation that the finance and pharma tables are realistic in any measurable way (value distributions, semantic consistency, join patterns, or similarity to real data). The healthcare corpus is an anchor because it starts from real MIMIC-III base tables, but the fully synthetic finance and pharma corpora have no such anchor. Without a comparison between generated tables and real domain-specific data, the poor performance of discovery methods on LAKEGEN corpora could be an artifact of synthetic data rather than evidence of genuine difficulty. A focused realism study, even on a small set of real finance/pharma tables, would significantly strengthen the claim.","section":"Sections 1, 3.2, 5"}],"minor_comments":[{"comment":"The formal definition of column joinability is garbled: 'cR ▷ ◁h(cR)=cS ∨cR=h(cS ) cS)̸=∅' is not a well-formed formula. Please rewrite it using a clear mapping function and a non-empty intersection condition.","section":"Section 2"},{"comment":"The column layout of Table 1 is misaligned, making it difficult to determine which checkmarks correspond to which column. The 'Human-annotated' checkmark for LAKEGEN is especially confusing given Section 3.1; please clarify or remove it.","section":"Table 1"},{"comment":"'Jaccard-Levenshiten' should be 'Jaccard-Levenshtein'.","section":"Tables 3, 4, 6 and text"},{"comment":"There is a typo in the sentence 'how the tables should are generated according to the inter-column dependencies'—it should read 'should be generated'.","section":"Section 3.2"},{"comment":"'Gosalse' should be 'Gosales'.","section":"Table 6"},{"comment":"The table header is confusing: it mixes ratio notation ('Non-exact join pairs/All semantic join pairs') with raw counts and per-method correct-prediction counts. Please clarify that the method columns show the number of correct predictions.","section":"Table 5"},{"comment":"The claim that semantic joinability drops F1 'up to 10%' is not consistently supported by Table 3; for example, COMA Healthcare increases from 0.68 (exact) to 0.75 (semantic). Please quantify where the drop actually occurs.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be an industry-conference submission (AMLC 2023) posted to arXiv in 2025. The 'Human-annotated' checkmark in Table 1 should be verified with the authors; if no human annotation was performed, it is a factual misrepresentation that must be corrected. The paper also does not provide a code or data repository, which would be valuable for a benchmark-generation contribution. The central gap—lack of label-quality validation for semantic joins—is fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on LAKEGEN. The core idea is genuinely new: use a domain ontology to derive table schemas, use an LLM to populate them, then use LLM-based semantic perturbations to create joinable pairs that are semantically equivalent but not identical. That combination doesn't exist in the cited corpora or generators. The paper also ships three domain-specific corpora (finance, pharma, healthcare) with joinability labels, and the experiments show a consistent pattern: four established matching methods score much lower on LAKEGEN than on Gosales/TPC-H/TPC-DS. That result is worth knowing.\n\nWhat the paper does well: the pipeline is plausible and the evaluation is internally consistent. The split into 'difficult' and 'easy' non-exact joinable pairs (Table 5) is a useful diagnostic, and the use of real ontologies (FIBO, DrugBank, MIMIC-III) gives the corpora a domain grounding that GitTables and Valentine lack.\n\nThe soft spots are real, and one is load-bearing. The ground truth for semantic joins is generated by the same LLM that produces the perturbations, and there is no systematic check that the perturbed values actually preserve semantics. The paper gives one anecdote (street address abbreviation) and then assumes text-davinci-003 avoids semantic breaks. Without a human-annotation study of a sample of generated pairs, the low F1 scores could be due to label noise. That's an internal validity problem, not just an external realism question. Relatedly, the abstract's framing that existing corpora 'do not contain human-annotated pairs' suggests LAKEGEN provides human-annotated ground truth, but it doesn't—the labels come from the ontology and perturbation pipeline. The Introduction also claims Section 3.2 shows LLMs 'score equally well' at generating domain-specific datasets, but that section contains no such comparison.\n\nMinor issues: no code or data released, and no head-to-head comparison with other LLM tabular generators like GReaT for the generation step. These matter less than the label validation but should be on the revision list.\n\nBottom line: this is a promising contribution that a serious referee should engage with. The central idea is sound, the flaw is fixable, and the benchmark artifacts would be useful to the community. I'd recommend peer review, with the expectation of major revisions: validate the semantic equivalence on a sample, release the artifacts, and soften the overclaims. If they do that, it becomes a solid benchmark paper.","headline":"A genuinely new pipeline for generating domain-specific tabular benchmarks, but the central claim of benchmark hardness rests on unvalidated LLM-generated ground truth.","tokens_in":14200,"tokens_out":4049,"would_cite":true,"duration_ms":38869,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LAKEGEN shows that ontologies plus large language models can produce domain-specific tabular corpora whose joinability relationships challenge current dataset discovery methods.","keywords":["dataset discovery","joinability discovery","tabular benchmark generation","large language models","ontology","semantic join","synthetic tabular data","data lake"],"falsifier":"Collect real-world domain-specific tables with human-annotated joinable pairs (for example from corporate data lakes or curated public sources), run the same four discovery methods on both the real pairs and LAKEGEN-generated pairs, and check whether the performance gap persists; if methods perform well on real pairs but poorly on synthetic ones, the benchmark's challenge does not transfer.","tokens_in":13244,"feed_emoji":"📊","tokens_out":3632,"duration_ms":38411,"temperature":0.7,"pith_summary":"The paper tries to establish that a combination of an ontology, an LLM, and semantic perturbations can generate realistic, domain-specific tabular benchmarks for evaluating dataset discovery methods. It argues that existing corpora—mostly subsets of open data with split-based synthetic ground truth—are too generic and lack the semantic complexity needed to stress test methods. LAKEGEN generates finance, pharma, and healthcare corpora, and the reported experiments show that current joinability and union search methods perform far worse on these corpora than on traditional benchmarks. If this is right, LAKEGEN provides a practical path to building hard, domain-specific test sets and reveals that modern discovery methods, especially embedding-based ones, struggle on semantic joins.","feed_headline":"Ontologies plus LLMs produce harder join-search benchmarks","feed_subtitle":"Generated finance, pharma, and healthcare corpora trip up four state-of-the-art joinability discovery methods.","key_machinery":"The central machinery is the ontology-to-table pipeline: each ontology concept becomes a table, each data property becomes a column, and each object property becomes a joinable column relationship. An LLM generates row values conditioned on dependent columns, and a set of perturbation functions (column-name cryptification, value synonyms, numeric noise, vertical and horizontal splitting) expands the corpus and creates semantically joinable pairs with divergent headers and cell contents.","core_discovery":"LAKEGEN is claimed to be the first corpus generator to combine ontologies, LLMs, and semantic perturbations for tabular data discovery evaluation. The method maps ontology concepts to table schemas and object properties to joinable column relationships, then prompts an LLM to populate tables row by row while preserving cross-column dependencies. Semantic perturbations—cryptified column headers, synonyms, numeric jitter, and table splits—create non-exact but semantically joinable pairs. The paper reports that, across three domains, the generated corpora are more challenging than existing public benchmarks: semantic join discovery is hard, and methods that rely only on learned column embeddings perform especially poorly.","pith_inferences":["Editorial inference: The paper demonstrates that LLM-generated corpora are harder, but it does not show they are realistic proxies for real data lakes; a natural next step is validating against human-annotated real-world joinable pairs in the same domains. ","Editorial inference: The same pipeline could be extended beyond joinability to other discovery tasks such as union search, semantic data type detection, and table summarization, since the generated tables carry domain semantics at both schema and value levels. ","Editorial inference: The perturbation functions are described as modular, so testable variations could be tuned to simulate different levels of schema drift and value noise, making it possible to measure how method performance degrades as semantic distance increases. ","Editorial inference: The paper does not report releasing the generated corpora; public release would let other groups reproduce the difficulty rankings and test new methods against the same benchmarks."],"forward_implications":["Domain-specific corpora can be produced with minimal human effort by drawing on existing ontologies such as FIBO, DrugBank, and MIMIC-III, giving dataset discovery researchers benchmarks for finance, pharma, and healthcare. ","Current discovery methods, including COMA, Jaccard-Levenshtein, Similarity Flooding, and Starmie, are not reliably stress-tested by generic open-data corpora; LAKEGEN-generated corpora expose particular weakness on semantically joinable pairs with different headers and values. ","The difficulty of semantic join discovery suggests that methods should combine traditional cell-overlap and schema signals with learned column representations rather than rely on embeddings alone. ","Ontology-driven generation can supply ground-truth joinability labels without the need for human annotation, avoiding a major bottleneck in building discovery benchmarks. ","The modular perturbation functions allow researchers to control difficulty by varying schema and value transformations, which could support training and hyperparameter tuning of discovery models before deployment."],"supporting_citations":[{"why":"Valentine supplies the split-based ground truth approach and the exact-joinability baseline methods that LAKEGEN contrasts with and extends.","marker":"[6]"},{"why":"GitTables is the example of an open-data corpus that lacks domain-specific semantic complexity and ground truth annotations, motivating LAKEGEN.","marker":"[8]"},{"why":"FIBO provides the finance ontology used to generate one of the three LAKEGEN corpora.","marker":"[17]"},{"why":"DrugBank supplies the pharma-domain ontology and semantic types used for the pharma corpus.","marker":"[24]"},{"why":"MIMIC-III is the source of the healthcare base tables that LAKEGEN perturbs to create the healthcare corpus.","marker":"[25]"},{"why":"MIMIC-III clinical database demo is used as the versioned source for the healthcare tables.","marker":"[26]"},{"why":"Starmie is the recent embedding-based discovery method included as a baseline in the evaluation.","marker":"[4]"},{"why":"COMA is a traditional schema-matching method used as a baseline for joinability prediction.","marker":"[23]"},{"why":"Similarity Flooding is another schema-matching baseline in the evaluation.","marker":"[27]"},{"why":"The idea that language models can serve as knowledge bases supports the paper's use of LLMs to generate domain-specific table values.","marker":"[16]"}],"fun_headline_variants":["LLM+ontology tables stump join-discovery methods","Generated corpora expose weaknesses in join search","LAKEGEN: tougher join benchmarks from LLMs and ontologies","LLM-generated data lakes challenge join-discovery methods","Semantic perturbations create harder joinability tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that tables and joinability ground truth produced by an LLM from an ontology are realistic stand-ins for domain-specific data lakes, so that poor method scores reflect genuine weaknesses rather than artifacts of synthetic data.","fun_headline_variants_meta":{"raw":{"variants":["LLM+ontology tables stump join-discovery methods","Generated corpora expose weaknesses in join search","LAKEGEN: tougher join benchmarks from LLMs and ontologies","LLM-generated data lakes challenge join-discovery methods","Semantic perturbations create harder joinability tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000746,"raw_usage":{"total_tokens":3304,"prompt_tokens":905,"completion_tokens":2399,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":2325}},"tokens_in":521,"tokens_out":2399,"duration_ms":17489,"temperature":1.0,"reasoning_tokens":2325,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:41:59.442875+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect real-world domain-specific tables with human-annotated joinable pairs (for example from corporate data lakes or curated public sources), run the same four discovery methods on both the real pairs and LAKEGEN-generated pairs, and check whether the performance gap persists; if methods perform well on real pairs but poorly on synthetic ones, the benchmark's challenge does not transfer.","supporting_citations":[{"cited_title":"Valentine: Evaluating matching techniques for dataset discovery","cited_arxiv_id":null,"evidence_quote":"Valentine supplies the split-based ground truth approach and the exact-joinability baseline methods that LAKEGEN contrasts with and extends."},{"cited_title":"GitTables: A Large-Scale Corpus of Relational Tables","cited_arxiv_id":"2106.07258","evidence_quote":"GitTables is the example of an open-data corpus that lacks domain-specific semantic complexity and ground truth annotations, motivating LAKEGEN."},{"cited_title":"The financial industry business ontology: Best practice for big data.Journal of Banking Regulation, 14(3-4):255–268, 2013","cited_arxiv_id":null,"evidence_quote":"FIBO provides the finance ontology used to generate one of the three LAKEGEN corpora."},{"cited_title":"Drugbank 5.0: a major update to the drugbank database for 2018.Nucleic acids research, 46(D1):D1074–D1082, 2018","cited_arxiv_id":null,"evidence_quote":"DrugBank supplies the pharma-domain ontology and semantic types used for the pharma corpus."},{"cited_title":"Mimic-iii clinical database demo (version 1.4).PhysioNet, 2019","cited_arxiv_id":null,"evidence_quote":"MIMIC-III clinical database demo is used as the versioned source for the healthcare tables."},{"cited_title":"Semantics-aware dataset discovery from data lakes with contextualized column-based representation learning.PVLDB, 2023","cited_arxiv_id":null,"evidence_quote":"Starmie is the recent embedding-based discovery method included as a baseline in the evaluation."},{"cited_title":"Coma—a system for flexible combination of schema matching approaches","cited_arxiv_id":null,"evidence_quote":"COMA is a traditional schema-matching method used as a baseline for joinability prediction."},{"cited_title":"Similarity flooding: A versatile graph matching algorithm and its application to schema matching","cited_arxiv_id":null,"evidence_quote":"Similarity Flooding is another schema-matching baseline in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The idea that language models can serve as knowledge bases supports the paper's use of LLMs to generate domain-specific table values."}],"review_version":1}