{"id":"6bde4012-6704-48f2-966f-9bd2ba1fc55c","arxiv_id":"2509.00728","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A structured review of open dataset search across tabular, spatial, JSON, graph, and vector data, plus the two-way relationship with LLMs.","lead":"This paper surveys how people find and reuse publicly available datasets, and how large language models are changing that process. It organizes recent methods by data type, from spreadsheets to maps to graphs, and outlines open problems like privacy, benchmarks, and cross-modal discovery.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 5.3 does not supply evidence for the claimed 'dataset search for LLM' direction: the RAG case studies retrieve documents/corpora, not open datasets, so the mutual-benefit claim is only half supported.","rationale":"The survey has real value: the modality-organized sections are informative, the LLM-for-dataset-search side is well supported, and the future-directions list is concrete. The reader's identified weakness—selection completeness and neutrality—is plausible and worth addressing. However, my stress-test identifies a more load-bearing problem internal to the central claim: Section 5.3's 'Dataset Search for LLM' half does not actually cite dataset search enabling RAG. The RAG applications listed in Table 7 are retrieval over documents, knowledge graphs, or domain corpora, not over open datasets, and the data-selection discussion mostly cites generic data-selection techniques rather than dataset-search systems. Because the abstract explicitly promises 'mutually beneficial' support in both directions, this unsupported direction weakens the central thesis. The fix is feasible: either add concrete dataset-backed RAG examples or soften the claim. The reader's verdict of CONDITIONAL remains appropriate; I do not see grounds to move to ACCEPT or REJECT. My conclusion therefore does not change the reader's verdict, though the condition should be revised to include this evidence gap, not only completeness and neutrality.","tokens_in":33824,"tokens_out":3828,"duration_ms":50992,"concrete_test":"Audit every reference in §5.3 and Table 7: classify the external retrieval source of each RAG application as (a) an open dataset per Definition 1.1, (b) a document/corpus/knowledge graph, or (c) other; and determine whether the retrieval step uses any dataset-search system from §§3–4. If zero of the RAG case studies retrieve open datasets from a dataset repository, and if Wang et al. [138] is the only data-selection work that actually integrates dataset search, then the abstract's 'in turn... enabling more effective integration into RAG frameworks' claim is unsupported and should be revised to a research opportunity or backed by new examples.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and §1.3 stake the survey's central contribution on a 'mutually beneficial relationship' between LLMs and open dataset search, including the direction that 'advances in dataset search can support LLMs by enabling more effective integration into RAG frameworks and data selection processes.' This direction is the weakest load-bearing point. In §5.3, after noting that RAG's retrieval component aligns with dataset search, the paper explicitly says it 'instead showcases recent applications of RAG across diverse domains.' The 15 cited RAG applications in Table 7 (e.g., Self-BioRAG, GraphQA, InstructRAG) retrieve scientific papers, knowledge graphs, or domain-specific text corpora—not open datasets as defined in Definition 1.1 and not through any dataset-search system surveyed in §§3–4. No cited example shows a dataset-search engine (e.g., Auctus, DBF, Starmie, LOTUS) supplying the retrieval source of a RAG pipeline. Similarly, the data-selection subsection cites generic data-selection works (LESS, Dolma, MateS) plus only one method (Wang et al. [138]) that actually integrates query-driven dataset search into a selection pipeline. Thus the claimed reverse direction is asserted rather than evidenced by the reference set. This is an internal evidence gap, not merely a completeness or neutrality concern: even a broader reference list would need to supply examples of dataset-backed RAG, not just RAG over text corpora.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This survey reviews recent research on open dataset search, moving beyond keyword- and metadata-based retrieval. It organizes the field by data modality (tabular, spatial, JSON, graph, and vector datasets), surveys query-by-example and natural-language search techniques, and discusses the interplay between LLMs and dataset search. The claimed central contribution is a two-way relationship: LLMs improve dataset search through query understanding, semantic planning, and interactive guidance, while dataset search supports LLMs through retrieval-augmented generation (RAG) and data selection. The survey also identifies open problems, including privacy-preserving search, task-oriented integration, cross-modal discovery, federated search, and benchmarks.","tokens_in":34157,"tokens_out":5461,"duration_ms":71716,"significance":"If the reference selection is representative and the reverse-direction claims are adequately evidenced, this survey would fill a useful gap by providing a modality-aware, LLM-centric overview of open dataset search. The paper's strengths are its broad taxonomy, formal definitions, reproduction of key equations (e.g., MaxSim, Hausdorff distance, JSON tree edit distance, GED/MCS), comparative tables that summarize representative methods, and an explicit roadmap of open challenges. It also makes the useful expository choice of distinguishing LLM-for-search from search-for-LLM. However, as a survey, its conclusions rest on the accuracy of its secondary summaries and on the completeness and neutrality of its reference selection, and neither can be fully verified from the manuscript. The most load-bearing weakness is the reverse-direction chapter, whose RAG examples do not actually demonstrate dataset-search-backed RAG.","major_comments":[{"comment":"The claimed direction 'advances in dataset search can support LLMs by enabling more effective integration into RAG frameworks' is not supported by the evidence in this subsection. After noting that RAG's retrieval component 'naturally aligns' with dataset search, the text explicitly states that the subsection 'instead showcases recent applications of RAG across diverse domains.' Table 7 lists 15 RAG applications (e.g., Self-BioRAG, GraphQA, InstructRAG), but these retrieve scientific papers, knowledge graphs, or domain-specific text corpora—not open datasets as defined in Definition 1.1, and not through any dataset-search system surveyed in Sections 3–4. None of the cited examples shows a dataset-search engine (e.g., Auctus, DBF, Starmie, LOTUS) supplying the retrieval source of a RAG pipeline. The abstract and Section 1.3 stake a core contribution on this mutual-benefit claim, so the re","section":"§5.3, Table 7"},{"comment":"The data-selection discussion similarly overstates the integration of dataset search into LLM data pipelines. The paragraph cites generic data-selection methods (LESS, Dolma, MateS) and then identifies only one method—Wang et al. [138]—that actually integrates query-driven dataset search into a selection pipeline. The claim that 'recent advances have begun to integrate dataset search into data selection pipelines' is supported by a single concrete example. Given that Section 6.2 itself mentions other systems (DeepResearchGym, STARK, GPT-Instructor) that embed dataset retrieval in end-to-end pipelines, the authors could strengthen this subsection substantially by moving those examples here or by adding additional published cases. Without that, the reverse-direction evidence remains one example, which is not enough to sustain the survey's broader 'mutually beneficial relationship' framing.","section":"§5.3, Data Selection for LLM"}],"minor_comments":[{"comment":"The 'Vector' row lists '[99]' (BioVSS) as a data source, but [99] is a search method, not an open data repository. The underlying Microsoft Academic Graph [127] appears to be the actual source. Please correct the citation or rephrase the row.","section":"Table 1"},{"comment":"Definition 1.2 covers keyword and exemplar queries but not natural-language queries, which become a major theme in Section 5.2. Consider extending the definition or adding a remark that NL queries are handled as a separate query type later in the survey.","section":"Definition 1.2"},{"comment":"The vector-dataset-search section includes several systems originally designed for passage/document retrieval (COLBERT, PLAID, SLIM, COIL, CITADEL, XTR). Since the survey's title and abstract are about dataset search, a brief justification of why token-vector collections over documents are treated as vector datasets would help readers accept this categorization.","section":"§4.1"},{"comment":"Figure 2 is dense and is referenced only once. Adding pointers from the pipeline stages to the corresponding sections (e.g., query mechanisms, similarity calculation, indexing) would improve navigability.","section":"Figure 2"},{"comment":"The discussion of spatial dataset search states that the field lacks a standardized benchmark, but no reference is given for this claim. A citation or a brief explanation of which evaluation gaps exist would strengthen the point.","section":"§4.2"},{"comment":"In text immediately before Table 7, the phrase 'retrieval-augmented generation' and the surrounding discussion could more precisely distinguish between retrieving documents/corpora and retrieving datasets. As written, the section's scope drifts from dataset search to general RAG.","section":"§5.3"}],"recommendation":"major_revision","confidential_remarks":"The main issue is not the survey's organization or the quality of its tables, which are generally solid, but the asymmetry between the two directions of the claimed LLM–dataset-search synergy. The reverse direction is under-evidenced, and the one concrete integration example in §5.3 (Wang et al. [138]) is from a co-author's group. It would be prudent to ask the authors to add independent examples of dataset-backed RAG or data selection, or to soften the claim to a research opportunity. Additionally, while the paper cites many prior surveys, several of the authors' own papers appear in positions that shape the modality taxonomy (e.g., [99], [154], [155], [156]); this is not improper for a survey, but the editors may wish to check that the selection is not unduly skewed toward the authors' systems."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuinely useful map of dataset search, organized by modality—tabular, spatial, JSON, graph, vector—with clear definitions, reproduced equations, and handy summary tables. Second, the paper's headline claim of a \"mutually beneficial relationship\" between LLMs and dataset search is only half supported: the LLM-for-search direction is well evidenced, but the search-for-LLM direction rests on assertion.\n\nWhat's actually new: explicit coverage of JSON document search and vector dataset search, which prior surveys (Chapman 2020, Paton 2023) skipped, and a two-way framing that treats LLMs as both a tool for and a beneficiary of dataset search. The survey is also honest about gaps—missing benchmarks for spatial and vector modalities, privacy-preserving search, federated search—and it does a solid job of summarizing the main techniques with enough detail to orient a newcomer. The reproduction of equations (MaxSim, Hausdorff, EMD, TED, GED) is correct as far as I can tell.\n\nThe soft spots are real but fixable. The biggest is Section 5.3. The paper claims advances in dataset search support LLMs through RAG, but then explicitly says it \"instead showcases recent applications of RAG across diverse domains.\" Table 7's fifteen examples retrieve papers, knowledge graphs, or domain text—not open datasets in the sense of Definition 1.1, and none use the dataset search systems surveyed in Sections 3–4. The data-selection subsection is thinner: only one cited method (Wang et al. [138]) actually integrates query-driven dataset search into a selection pipeline. So the reverse direction is not evidenced by the reference set. The stress-test note lands; a revision needs to either supply real dataset-backed RAG examples or reframe the claim as a research opportunity.\n\nTwo minor concerns. The survey calls itself \"systematic and comprehensive\" but describes no selection methodology, which makes the taxonomy hard to audit. And the citation base leans noticeably on the authors' own spatial and vector work ([97], [154], [155], [156], [99], [168]). Neither is fatal, and self-citation in these areas is partly justified because they're the authors' specialty, but a broader reference set would be more persuasive.\n\nBottom line: this is a competent survey that deserves a serious referee, with the expectation of a revision to fix the Section 5.3 gap and disclose how references were selected. I'd bring it to a reading group for anyone working in data discovery or RAG infrastructure.","headline":"A useful modality-aware survey of dataset search that mostly delivers, but the 'dataset search for LLM' half of its central pitch is asserted rather than shown.","tokens_in":34604,"tokens_out":1991,"would_cite":true,"duration_ms":22948,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey argues that open dataset search has outgrown keyword matching and is now defined by a two-way partnership with large language models.","keywords":["open dataset search","large language models","query-by-example","content-aware retrieval","retrieval-augmented generation","data selection","multimodal datasets","data discovery"],"falsifier":"A systematic sweep of published dataset search work that finds a material body of research on a modality the survey does not index (e.g., time-series or audio datasets), or that shows the LLM-for-search systems the survey cites are not adopted or cited by downstream RAG or data-selection work, would weaken both the completeness of the modality taxonomy and the claim that the LLM-dataset search relationship is genuinely two-way.","tokens_in":33760,"feed_emoji":"🤝","tokens_out":7903,"duration_ms":90313,"temperature":0.7,"pith_summary":"The paper tries to establish that open dataset search has moved beyond metadata and keyword matching into content-aware, example-based retrieval, and that this shift is best understood by organizing the field by data modality. It reviews tabular, spatial, JSON, graph, and vector datasets, cataloging how each modality defines similarity and accelerates search. Its central claim is that large language models and dataset search are mutually beneficial: LLMs improve query understanding, semantic matching, and interactive guidance, while dataset search supplies the high-quality, task-relevant data that retrieval-augmented generation and data selection depend on. If this framing holds, future research and products will converge on modality-aware, LLM-orchestrated dataset discovery rather than on traditional keyword portals. The survey also maps open problems such as privacy-preserving search, cross-modal discovery, federated search, and missing benchmarks, arguing these are the next bottlenecks.","feed_headline":"Large language models and dataset search now feed each other","feed_subtitle":"A survey maps how example-based, content-aware discovery spans five data modalities and powers AI data pipelines.","key_machinery":"The organizing instrument is a modality taxonomy: tabular, vector, spatial, JSON document, and graph datasets, each with a formal definition (for example, a vector dataset is a set of embedding vectors, a spatial dataset is a set of georeferenced points, a JSON document is a nested key-value tree, and a graph dataset is a vertex-edge pair). The paper maps each modality to its dominant similarity measure—MaxSim for vector sets, Earth mover's distance and Hausdorff distance for spatial data, tree edit distance for JSON, and graph edit distance or maximum common subgraph for graphs—along with indexing and acceleration techniques. Over this taxonomy it overlays the two-way LLM relationship, trea","core_discovery":"The paper's contribution is a structured synthesis: modern open dataset search is no longer a metadata-matching problem but a content-aware, modality-specific retrieval problem, and the most forward-looking direction is the two-way coupling between LLMs and dataset search. On one side, LLMs automate dataset construction, cleaning, and transformation, and enable natural-language queries, semantic planning, and relevance estimation. On the other side, dataset search supports LLMs by selecting datasets for retrieval-augmented generation and for data selection during pretraining, fine-tuning, and in-context learning. The survey develops this thesis across five data modalities, offering formal de","pith_inferences":["If the mutual-benefit thesis holds, dataset search could become a closed-loop controller for LLM training: a model's judged failures could seed queries for corrective datasets, and the retrieved data could then be evaluated by the model, forming a feedback loop; the paper mentions agentic integration but does not formalize this loop.","The modality taxonomy suggests that cross-modal dataset search is the natural stress test: a shared embedding space for tables, JSON trees, graphs, and spatial points would unify the separate similarity measures, and a cross-modal join and union benchmark would directly test the value of such a unified representation.","The paper's emphasis on content over metadata implies that search quality may be robust to missing or poor metadata; a direct experiment comparing ranking quality on datasets with full metadata versus deliberately stripped metadata would quantify how much content signals compensate.","The recurring use of sketch and hash approximations across tabular, vector, and spatial search suggests a common abstraction—set containment at scale—that could be factored into a shared data-discovery index, potentially simplifying future system design."],"forward_implications":["Example-based and natural-language queries will increasingly replace keyword-only interfaces in dataset search systems.","LLM-based schema inference, cleaning, and transformation should be treated as part of the dataset search pipeline, not as separate data-engineering tasks.","Dataset search will become a core stage in LLM development, feeding retrieval-augmented generation and data-selection pipelines with queryable, filterable data.","Standardized benchmarks are missing for spatial and vector dataset search, and quality control is not yet integrated into search; closing these gaps is a prerequisite for practical progress.","Privacy-preserving similarity calculation, indexing over encrypted data, and federated search remain open problems that need new index and acceleration designs."],"supporting_citations":[{"why":"Supplies the dominant metadata-based engine that the paper positions as the traditional baseline to be surpassed.","marker":"[18]"},{"why":"Prior survey on dataset search that establishes the baseline coverage this survey extends beyond metadata and keywords.","marker":"[24]"},{"why":"Prior tabular-focused survey of table discovery; one of the comparison baselines in Table 2.","marker":"[45]"},{"why":"Prior survey of dataset discovery and exploration; another comparison baseline the survey differentiates itself from.","marker":"[117]"},{"why":"Closest predecessor covering LLMs for tabular data discovery, which the survey extends to a full multimodal and two-way LLM perspective.","marker":"[54]"},{"why":"Defines the vector dataset search problem as set-of-embeddings retrieval, providing the framing for Section 4.1.","marker":"[43]"},{"why":"Introduces Earth mover's distance for spatial dataset search, a load-bearing similarity example in Section 4.2.","marker":"[156]"},{"why":"Supplies the MaxSim late-interaction operator used as the canonical vector-set similarity measure.","marker":"[86]"},{"why":"Provides the survey of data selection for language models that grounds the 'dataset search for LLM' half of the thesis.","marker":"[4]"},{"why":"Shows dataset search used inside a data-selection pipeline, the concrete instance of the reverse direction in the mutual-benefit argument.","marker":"[138]"}],"fun_headline_variants":["LLMs and dataset search now feed each other","How LLMs supercharge dataset search, and vice versa","The mutual improvement loop of LLMs and dataset search","Survey: content-aware dataset search enters LLM era"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The survey assumes that the research landscape is accurately represented by its modality-based partition and by the particular selection of papers, so that the taxonomy and the open-problems list correspond to the field's true shape rather than to the authors' reading of it.","fun_headline_variants_meta":{"raw":{"variants":["LLMs and dataset search now feed each other","How LLMs supercharge dataset search, and vice versa","The mutual improvement loop of LLMs and dataset search","Survey: content-aware dataset search enters LLM era"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"total_tokens":1472,"prompt_tokens":787,"completion_tokens":685,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":622}},"tokens_in":531,"tokens_out":685,"duration_ms":7423,"temperature":1.0,"reasoning_tokens":622,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:16:09.562400+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A systematic sweep of published dataset search work that finds a material body of research on a modality the survey does not index (e.g., time-series or audio datasets), or that shows the LLM-for-search systems the survey cites are not adopted or cited by downstream RAG or data-selection work, would weaken both the completeness of the modality taxonomy and the claim that the LLM-dataset search relationship is genuinely two-way.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior survey of dataset discovery and exploration; another comparison baseline the survey differentiates itself from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Earth mover's distance for spatial dataset search, a load-bearing similarity example in Section 4.2."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows dataset search used inside a data-selection pipeline, the concrete instance of the reverse direction in the mutual-benefit argument."}],"review_version":1}