{"id":"c9941c3d-d4a9-44a2-b0d6-b75b94dfc395","arxiv_id":"2608.04536","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Finding columns that operationalize a broad research question is harder than standard column search or schema linking; on the new OADD-Bench, the best agent achieves only 0.465 recall at a 5x budget.","lead":"This paper defines a database search task in which a researcher starts with a broad concept, such as social isolation, and must find the columns that can measure it even when no field has a matching name. It introduces a benchmark built from 111 published studies and shows that the best tested system finds fewer than half of the relevant columns, leaving the problem open.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Recall@R scores only the one operationalization used in each source paper; a system returning a different defensible measure scores zero, so the 'systems fail' claim may overstate the OADD gap.","rationale":"The reader's weakest assumption targets the LLM-generated labels and limited human verification. That is a real concern, but I find a more load-bearing issue in the evaluation design: the benchmark equates 'success' with recovering the exact column set of one published study, even though the problem definition permits any scientifically defensible operationalization. This affects the central empirical claim directly. If a system discovers a different valid measurement, it is scored as a failure, so low recall does not by itself prove that current systems fail at OADD. The paper does not report precision or any check for alternative valid answers, so the reader cannot tell whether the 0.465 recall means the task is genuinely open or simply that the paper's specific choices are hard to guess. The concrete test I propose would separate these interpretations: if expert-generated operationalizations overlap the agent's outputs substantially, the benchmark's single-target scoring understates system capability. This does not invalidate the paper's conceptual contribution or the benchmark as a reproducible resource, but it does mean the headline 'open problem' claim needs qualification. The reader's conditional verdict remains appropriate; no change in verdict is needed, hence UNCHANGED.","tokens_in":10457,"tokens_out":4445,"duration_ms":57817,"concrete_test":"Sample 20 benchmark questions and have domain experts (or a second, independently prompted LLM from a different family) produce their own defensible operationalizations directly from HRS documentation, without seeing the paper's target sets. Score the GPT-5.5 agent's outputs against these expert sets for both recall and precision. If recall against expert sets is comparable to the reported 0.465, the low scores reflect benchmark specificity; if recall is substantially higher, the 'systems fail' claim is an artifact of scoring only the paper's specific measurement choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The problem definition (Section 2.1) allows any 'scientifically defensible operationalization.' Section 2.2 acknowledges that 'other defensible measurements may exist,' yet every benchmark label is the column set from a single published study's reported measurement. Section 4 scores exact-column recall against exactly that one set and reports no precision or alternative-validity evaluation. Consequently, a system that returns a coherent, defensible bundle that a domain expert would endorse receives recall 0 if it does not overlap the paper's particular columns. The conclusion that 'OADD remains an open problem' (Section 6) therefore depends on an untested assumption: that the space of defensible operationalizations is narrow enough that missing the paper's set implies failing the task. This concern is distinct from label quality: even if all labels were perfectly correct, low recall against one paper-specific target would not establish that systems cannot operationalize the concept. The paper's own Table 2 reports only recall, so high-precision alternative answers and pure garbage are indistinguishable in the evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces operationalization-aware data discovery (OADD), a data-discovery task in which a broad research question must be mapped to concrete database columns that jointly form a scientifically defensible measure. It constructs OADD-Bench from Health and Retirement Study (HRS) papers by using an LLM pipeline to extract questions and reconstruct paper-reported measurements, grounding them to HRS identifiers. The paper evaluates lexical and dense retrieval, five schema-linking adaptations, and LLM-based agents under a question-only, paper-hidden protocol, reporting exact-column recall at R, 2R, and 5R. The central claim is that direct retrieval reaches at most 0.185 recall, the strongest schema-linking adaptation reaches 0.401, and the best OADD-directed agent reaches 0.465, showing that OADD remains an open problem.","tokens_in":10702,"tokens_out":5625,"duration_ms":61055,"significance":"The problem is real and well motivated; the motivating example and the formal definition in Section 2.1 are clear. The benchmark construction treats published papers as external evidence, and the release includes provenance JSON, scoring code, and HRS metadata, which are concrete strengths. The evaluation boundary is carefully documented: systems receive only questions, permitted years, and metadata, while source papers are hidden. If the benchmark labels and the single-target recall metric are accepted, the results would provide a valuable diagnostic showing that existing retrieval and schema-linking techniques are insufficient for operationalization-driven discovery. The main risks are the validity of the LLM-generated labels and the interpretation of exact-column recall against one paper-specific target; the paper is honest about some limitations but does not resolve them.","major_comments":[{"comment":"The benchmark labels are load-bearing, yet 140 of 160 examples are produced by the GPT-5.5 column miner with deterministic checks but no human gold verification; for the remaining 20, human annotations are explicitly not treated as gold. The reported macro set F1 of 0.933 on 640 labels implies roughly 7% disagreement between two reconstructions even on the checked subset. Since every recall number in Table 2 is computed against these labels, a systematic LLM bias in reconstructing measurements would directly masquerade as system failure. Please provide either a human gold verification on a larger sample, an error analysis comparing LLM-reconstructed mappings to the source papers, or a sensitivity analysis that rescore the benchmark with a verified subset to demonstrate that the recall gap is not an artifact of label generation.","section":"Section 3.4"},{"comment":"Recall@R, Recall@2R, and Recall@5R are computed against the single column set used in the source publication, but the problem definition in Section 2.1 asks for 'a scientifically defensible operationalization,' and Section 2.2 explicitly acknowledges that other defensible measurements may exist. A system that returns a different but defensible bundle scores zero if it does not overlap the paper's set, and no precision or alternative-validity evaluation is reported in Table 2. Thus low recall does not by itself establish the conclusion that OADD remains open: the evaluation conflates 'missing the paper's particular operationalization' with 'failing to operationalize the concept.' This concern is independent of label quality. Please add precision or alternative-validity analyses, such as expert judgments on randomly sampled outputs, or an evaluation that credits alternative defensible operationalizations.","section":"Section 2.3"},{"comment":"The same model family (GPT-5.5) is used both by the column miner that generates the benchmark labels and by the best-performing agent in Table 2. Although the construction pipeline uses paper-derived evidence and deterministic checks, shared priors between label generator and evaluator could inflate the OADD agent's recall relative to methods not built on the same model. I would like to see either an agent evaluation with a different model family or an analysis showing that benchmark labels are stable when the column miner is run with a different backbone. Without this, the GPT-5.5 agent's 0.465 recall should be interpreted cautiously.","section":"Section 3.3"}],"minor_comments":[{"comment":"Please clarify how the 20 annotated examples were selected and whether they overlap the 140 miner-produced examples or are additional examples; the current text says only that they come from distinct papers.","section":"Section 3.4"},{"comment":"Identifier formatting is inconsistent, e.g., 'OLB028B3' versus 'OLB028B01'; a uniform zero-padded scheme would avoid ambiguity.","section":"Table 1"},{"comment":"The sentence 'Across many questions, low recall at 5R means repeatedly missing fields already demonstrated to support the focal concepts' should be qualified, since low recall can also mean finding a different defensible operationalization rather than failing the task.","section":"Section 2.3"},{"comment":"Only the AutoLink-versus-agent difference is reported with a bootstrap confidence interval; the other headline comparisons in Table 2 are point estimates without uncertainty. Reporting CIs or variance estimates for the main comparisons would strengthen the conclusions.","section":"Section 4.2"},{"comment":"To support reproducibility, please include the exact prompt templates used for the question miner and column miner in the artifact or an appendix.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper fits a data-management or information-retrieval venue and the benchmark construction is substantial. The main risks are addressable: human-verified labels and an alternative-validity evaluation would make the central claim much stronger. I do not see grounds for rejection, but the benchmark's evidentiary basis needs strengthening before the reported recall numbers can be interpreted as evidence that OADD is an open problem."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Chen and Jagadish define operationalization-aware data discovery, a real and previously unnamed task: given a scientific question and a database, find the columns that can measure the concept, even when the columns do not resemble the query. The benchmark built from 111 published HRS papers is a genuinely useful artifact, and the separation between construction (paper-conditioned) and evaluation (question-only) is done carefully. That part is solid.\n\nThe main soft spot is the evaluation metric. Every label is the column set from one published study's measurement, and Recall@R scores against that exact set. The paper acknowledges other defensible measurements exist, but nothing in the evaluation rewards a system that returns a different coherent bundle. So the headline conclusion – that OADD remains open – is only established for reproducing the paper's particular operationalization, not for the task as defined. A system that does OADD well by any reasonable standard would get recall 0 if it picked a different yet defensible measure. This is a load-bearing validity gap, not a minor niggle.\n\nThe secondary concern is the labels. 140 of 160 examples have ground truth produced by a GPT-5.5 column miner, with no human gold check; the consistency check on 20 examples is explicitly not gold. The same model family powers the best agent. That is a real circularity risk, though anchored because the labels are tied to published descriptions and HRS documentation. The absolute recall numbers (0.465 at 5R) should be read as upper bounds rather than true failure rates.\n\nWhat is good: the paper reports a bootstrap CI for the key comparison, releases the benchmark and scoring code, and is unusually honest in the limitations section – they state human annotations are not gold, and the release is not a sample of all HRS studies. The abstention mechanism and the provenance tracking are careful engineering.\n\nWho is this for: data management and scientific data discovery researchers. It gives them a new task and a reusable evaluation artifact. It deserves peer review, but the authors should be asked to either measure alternative operationalizations (e.g., via expert judges) or soften the claim to 'systems fail to reproduce published operationalizations.' I would recommend conditional acceptance: the task formalization and benchmark are valuable, but the central empirical claim needs another pass.","headline":"A useful new task and benchmark, but the recall metric against a single published operationalization overstates the 'systems fail' conclusion.","tokens_in":11169,"tokens_out":2577,"would_cite":false,"duration_ms":29453,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper defines operationalization-aware data discovery, builds a benchmark from published studies, and shows that current retrieval and schema-linking methods cover under half of the correct measurement columns.","keywords":["operationalization-aware data discovery","column retrieval","benchmark construction","schema linking","large language models","Health and Retirement Study","measurement bundles","exact-column recall"],"falsifier":"Take a random sample of 50 of the 160 benchmark questions, have two independent domain experts with full database documentation identify the columns needed to measure each question, and measure their agreement with the benchmark labels; if the experts agree with the labels on fewer than about 60 percent of columns, or if the experts disagree with one another as much as they disagree with the labels, the ground truth is not stable enough to support the paper's conclusions. A second check: if a system given the source paper text were to retrieve the target columns at near-perfect recall, the difficulty would lie in reconstructing measurements from text rather than in discovering them from the question.","tokens_in":10277,"feed_emoji":"🔍","tokens_out":6906,"duration_ms":71907,"temperature":0.7,"pith_summary":"Researchers often approach a data repository with a broad question, such as whether social isolation affects healthy aging, and need columns that can measure that concept. The paper argues that useful columns usually do not resemble the question, and that the real task is operationalization: choosing which concrete fields, often in combination, provide a defensible measure. It defines this as operationalization-aware data discovery (OADD) and builds OADD-Bench, a benchmark of 160 questions and 4,682 question–column labels mined from 111 published studies of a large longitudinal survey, treating each paper as a record of how researchers actually measured their concepts. On this benchmark, direct retrieval reaches at most 0.185 recall, the strongest schema-linking adaptation reaches 0.401, and the best OADD-directed agent reaches 0.465, so the problem remains open.","feed_headline":"Best column search finds only 46.5% of measurement columns","feed_subtitle":"When a concept has no matching field, even the strongest agent covers under half the columns documented in published research.","key_machinery":"The 'operationalization gap' is the central object: the logical distance between a broad concept and the concrete observations that can measure it, where a useful column may matter only as part of a bundle. The paper's construction pipeline is the machinery that makes the gap measurable: a question miner reframes paper questions without revealing measurement details, a paper-conditioned column miner reconstructs measurements through coarse-to-fine retrieval, evidence-packet screening, and a grounding-or-abstain final pass, and deterministic checks reject unsupported identifiers. Evaluation then converts the gap into a score using exact-column recall at output budgets of one, two, and five times the target size.","core_discovery":"The paper's central claim is that OADD is a distinct, previously unstudied discovery problem: given a broad research question and a database, the correct answer is a set of columns that jointly realize a scientifically defensible measurement, even when no single column resembles the question. To make this concrete, the paper constructs OADD-Bench by treating empirical papers as records of schema in use: a question miner extracts and reframes paper-supported questions, a paper-conditioned column miner reconstructs the measurements and grounds them to exact database identifiers, and only mappings supported by the publication and database documentation are admitted. The resulting evaluation shows that all tested methods fall short: the strongest OADD-directed agent, with an output budget five times the target size, covers only 46.5 percent of the target columns, while direct retrieval covers at most 18.5 percent. The paper concludes that OADD is an open problem requiring methods that reason about measurement bundles rather than direct relevance.","pith_inferences":["If the operationalization framing is right, data discovery in scientific repositories should move from ranking individual columns to proposing measurement bundles with explanations, and evaluation should reward covering the components of a defensible measure.","A natural test of the framing's generality is to apply the same paper-grounded construction method to other longitudinal repositories; if the gap is much smaller there, the difficulty may be specific to the benchmark rather than intrinsic to operationalization.","Because 140 of the 160 label sets were produced by an LLM miner without human gold verification, a large human-gold annotation study would calibrate whether the reported recall numbers measure true discoverability or agreement with LLM reconstructions.","The grounding-or-abstain mechanism suggests that a deployable OADD system should be allowed to report that a database cannot support a question, rather than being forced to output a ranking."],"forward_implications":["OADD-Bench gives the data management community a reproducible target: systems must recover indirect measurement bundles, not just directly relevant columns.","Schema-linking mechanisms that optimize direct query–column relevance transfer poorly to OADD, so new methods need to model joint realizability of a measurement.","The performance gap is not an output-budget artifact: increasing the allowed outputs from twice to five times the target size barely changes recall, while the best agent's average output stays far below its allowance.","Within a fixed agent design, model quality strongly affects bundle discovery, with recall at five-times budget rising from 0.272 to 0.465 across the tested models.","The benchmark opens the way for interactive researcher validation, provenance-aware grounding, and discovery across repeated survey waves."],"supporting_citations":[{"why":"Supplies the database context: the longitudinal survey's cohort profile and structure used for grounding all column identifiers.","marker":"[22]"},{"why":"Provides the motivating example of social isolation measured by a bundle of household and social-participation variables.","marker":"[3]"},{"why":"Exemplifies a composite measure whose 15 components must be grounded to exact survey identifiers across multiple waves.","marker":"[20]"},{"why":"The strongest schema-linking adaptation baseline against which the OADD-directed agent is compared.","marker":"[31]"},{"why":"The released schema-item ranker used as a non-LLM schema-linking baseline in the evaluation.","marker":"[13]"},{"why":"The publication bibliography that defines the frame from which papers were crawled and selected for benchmark construction.","marker":"[10]"}],"fun_headline_variants":["Operationalization-aware discovery: new task, best recall only 46.5%","Even top agent finds under half of needed measurement columns","New benchmark: OADD agents cover at most 46.5% of targets","Column discovery for broad questions: best recall 46.5%","Operationalization-aware data discovery remains an open problem"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the benchmark labels correctly capture how the published papers actually measured their concepts, since the recall numbers compare systems against those labels; if the LLM-produced labels are biased or incomplete, the reported gap may overstate the difficulty of OADD.","fun_headline_variants_meta":{"raw":{"variants":["Operationalization-aware discovery: new task, best recall only 46.5%","Even top agent finds under half of needed measurement columns","New benchmark: OADD agents cover at most 46.5% of targets","Column discovery for broad questions: best recall 46.5%","Operationalization-aware data discovery remains an open problem"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000504,"raw_usage":{"total_tokens":2517,"prompt_tokens":1057,"completion_tokens":1460,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":673,"completion_tokens_details":{"reasoning_tokens":1368}},"tokens_in":673,"tokens_out":1460,"duration_ms":13331,"temperature":1.0,"reasoning_tokens":1368,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:34:28.482782+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of 50 of the 160 benchmark questions, have two independent domain experts with full database documentation identify the columns needed to measure each question, and measure their agreement with the benchmark labels; if the experts agree with the labels on fewer than about 60 percent of columns, or if the experts disagree with one another as much as they disagree with the labels, the ground truth is not stable enough to support the paper's conclusions. A second check: if a system given the source paper text were to retrieve the target columns at near-perfect recall, the difficulty would lie in reconstructing measurements from text rather than in discovering them from the question.","supporting_citations":[{"cited_title":"Faul, Mary Beth Ofstedal, Kenneth M","cited_arxiv_id":null,"evidence_quote":"Supplies the database context: the longitudinal survey's cohort profile and structure used for grounding all column identifiers."},{"cited_title":"Torres, Yulin Yang, Karla Renata Flores Romero, Mary C","cited_arxiv_id":null,"evidence_quote":"Provides the motivating example of social isolation measured by a bundle of household and social-participation variables."},{"cited_title":"Smith, and Ashwin A","cited_arxiv_id":null,"evidence_quote":"Exemplifies a composite measure whose 15 components must be grounded to exact survey identifiers across multiple waves."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The publication bibliography that defines the frame from which papers were crawled and selected for benchmark construction."}],"review_version":1}