{"id":"688f0525-4c9f-4f10-9fef-d815fff7e71d","arxiv_id":"2608.07946","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A verification-harness LLM system recovers column semantics from values alone, but its measured advantage over a direct LLM is selective abstention, not better prose, and a code-enforced gate makes that abstention backbone-independent.","lead":"This paper presents Rosetta, a system that guesses what cryptic database columns and values mean by looking only at the data, and refuses to guess when the evidence is too weak. It finds the system's real advantage is knowing when to stay silent, not writing better descriptions than a plain language model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Public-benchmark memorization is not controlled for the headline reconstruction and ICD-9 decode results, so the 'from values alone' attribution may not hold; the paper flags this only for the join-graph experiment.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the primary findings are measured on public BIRD and i2b2/Synthea data that a frontier LLM may have memorized, and the paper only flags this for the join-graph experiment. This is the single most load-bearing issue because the paper's central contribution is framed as recovering metadata 'from values alone,' and the headline numbers—Rosetta's 0.475 recall-when-claimed on 42% coverage, and the 95.5% ICD-9 decode rate—are the quantitative core of that claim. If the model has memorized the gold documentation or ICD-9 mappings, the attribution to value-derived reconstruction fails, and the external validation is not external. The ICD-9 decode is especially decisive because the value-to-meaning mapping is not derivable from the value's structure; the paper's own prefix control only eliminates one pathway, not memorization. The concern is concrete and testable: a contamination-controlled evaluation on private data would settle whether the results transfer. The paper's many honesty safeguards (registered predictions, negative results, backbone swaps, blinded judging) are real and reduce the risk that the selection/abstention behavior is fabricated, but they do not address the training-data contamination threat to the 'from values alone' attribution. The verdict should remain CONDITIONAL: the contribution is plausible and well-measured in many respects, but the headline attribution needs a contamination-controlled replication before the claim can be accepted without qualification. This is exactly the reader's conditional posture, so no change to the verdict is needed.","tokens_in":50383,"tokens_out":10890,"duration_ms":116218,"concrete_test":"Run the Section 5.2 reconstruction and Section 5.4.1 decode tasks on a private, never-published warehouse: (a) a schema with third-party-authored column names and descriptions not present on the public web, and (b) a proprietary code-to-meaning table (e.g., a random permutation of ICD-9 labels or a synthetic opaque code system). If recall-when-claimed and decode accuracy collapse or abstention rises sharply, the public-benchmark results are contaminated. For the decode task specifically, compare accuracy on real ICD-9 codes versus the same codes with meanings randomly permuted: if the system 'decodes' the permuted codes, it is using shape or heuristics; if it fails or abstains, the real-code success is memorized mapping, not value-derived reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim's external evidence rests on public BIRD and i2b2/Synthea data that a frontier LLM (Gemini 3.1 Pro) has plausibly memorized. The ICD-9 decode result (Section 5.4.1) is the clearest case: the value '493.90' contains no information about 'asthma'—the mapping must come from training memory. The prefix-stripping control (0.745 vs 0.740) rules out reading the answer off the 'ICD9:' prefix, but it does not rule out memorized code-to-meaning tables, which are ubiquitous on the web. The paper applies its own contamination caveat only to the names-visible join experiment (Section 5.2.10: 'these are public benchmark schemas a frontier model has plausibly seen in training'), not to the headline column-description reconstruction (Section 5.2) or the ICD-9 decode. The held-out BIRD train databases in Section 5.2.8 are still public BIRD data, so they do not resolve this. If the model recalls the gold documentation or ICD-9 mappings, the 'from values alone' framing is confounded: the absolute recovery numbers and the 'it decodes from values' claim are overstated, even if the relative selection advantage over a same-model baseline might survive. The abstract's 'decodes 95.5% of 134 real ICD-9 codes from values alone' is particularly vulnerable, especially because Section 5.4.1 reports the harness adds nothing on that task (plain model also 0.745).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Rosetta, a system that places an LLM inside a deterministic verification harness to reconstruct column semantics (expanded names, descriptions, value decodes) from values alone, and then uses the reconstructed catalog for calibrated abstention in text-to-SQL. It evaluates against BIRD human documentation with identifiers destroyed, against a blind i2b2/Synthea warehouse for ICD-9 decoding, and against Spider for query-time routing, comparing a statistical arm, an LLM-direct arm, and the full harness. The headline result is a selection effect: Rosetta commits to 42% of columns at 0.475 recall-when-claimed versus 0.223 on 94% for the LLM-direct baseline, with no prose advantage on the paired common ground, and a code-enforced gate removes ungrounded claims. The paper also reports backbone-swap results, a validated lexical metric, an honest set of negative results, and a frozen experimental record.","tokens_in":50575,"tokens_out":6832,"duration_ms":82029,"significance":"If the attribution holds, the contribution is significant: a competence detector that converts a naive model's confident silent errors into auditable refusals, with label-free confidence and a deterministic route decision. The paper is unusually careful methodologically: external ground truth, cluster bootstraps over databases, a blinded judge, a second rater, backbone swaps, registered predictions, held-out databases, and explicit negative results are all strengths that go well beyond typical practice. However, two load-bearing issues need to be addressed before the central claims can be accepted at face value: possible memorization of public benchmarks undermines the 'from values alone' attribution, and the paper's own ablation and backbone-swap results contradict the claimed mechanism for abstention in the default configuration.","major_comments":[{"comment":"The central claim that semantics are recovered 'from values alone' is not protected against the model's memorization of public benchmarks. BIRD mini-dev and i2b2/Synthea are public datasets, and Gemini 3.1 Pro is a frontier model whose training data plausibly includes these schemas, their human-authored documentation, and widely republished ICD-9 code-to-meaning tables. The only contamination caveat in the paper appears in Section 5.2.10, and it is applied to the names-visible join experiment ('these are public benchmark schemas a frontier model has plausibly seen in training'), not to the headline reconstruction study of Section 5.2 or the ICD-9 decode of Section 5.4.1. The prefix-stripping control (0.745 vs 0.740) rules out reading the answer off the 'ICD9:' prefix, but it does not rule out recalling memorized code-to-meaning tables; the identifier-destruction control only rules out use of column names. The held-out BIRD train databases in Section 5.2.8 are still public BIRD data. Since an ICD-9 code such as '493.90' contains no information about asthma, the 95.5% decode rate must be substantially attributed to parametric knowledge rather than to inference from the values. This does not necessarily destroy the relative selection advantage over an identical model used directly, but it weakens the 'from values alone' framing and the absolute recovery numbers. A concrete remedy would be an evaluation on a non-public corpus with novel value-to-meaning mappings, or at minimum an explicit disclosure that the headline results measure recovery under possible training-data contamination.","section":"5.2, 5.4.1, 5.2.10"},{"comment":"The paper's central claim states that the system's declining behavior is 'driven by the deterministic evidence rather than by the model's self-report' (Section 1 and the abstract). The paper's own measurements contradict this for the default configuration. Section 5.2.8 reports that on Claude Sonnet 4.6 arm C's coverage rises from 0.422 to 0.823 and the evidence-tracking gap is no longer detectable (-0.089, interval spanning zero); Section 5.2.9 shows that all three abstention behaviors (column, table, value) collapse on a second backbone because abstention is requested in a prompt rather than enforced in code. Section 5.8.1 further shows that authority features add no incremental discrimination and that the ladder caps rarely bind. The code-enforced gate of Section 5.2.8 does make no-evidence coverage 0.000, but this is by construction rather than a measured property of the deterministic evidence, and it was implemented after the fact, even if under a registered protocol. The central claim should be restated so that the default configuration is described as model-dependent selection and the gated configuration as enforced abstention; the current phrasing overstates what is established.","section":"1, 5.2.1, 5.2.8, 5.8.1"},{"comment":"The abstract credits Rosetta with decoding 95.5% of ICD-9 codes from values alone and abstaining on all 44 NDC drug codes. Section 5.4.1 states that 'the harness adds nothing here': the plain model achieves the same 0.745 overall decode rate and also declines all 44 NDC codes on the same backbone, and the abstention in the default configuration is prompt-requested rather than code-enforced. The headline therefore attributes to the architecture a behavior that the paper's own measurements attribute to the backbone. The abstract and the contribution list should be rephrased to credit the model for the decoding and to credit the harness only for provenance and the audit trail, or the gate-enabled variant should be used if the NDC refusal is to be presented as a system property.","section":"Abstract, 5.4.1"}],"minor_comments":[{"comment":"The destroyed-condition row in the join-graph table contains the slash-separated entry 'stems / naked model — / 0.000', which is confusing to read as a single cell; separate rows or explicit column headers would make the 0.000 precision/recall for both proposers clearer.","section":"5.2.10 (Table)"},{"comment":"The abstract says 799 columns and Section 5.2 reports 680 paired columns across eleven databases; the relationship between the full mini-dev set and the paired subset is explained, but the abstract's '680 paired columns' should make clear that recovery numbers are conditional on the paired subset, since the unpaired columns affect coverage denominators.","section":"Abstract and 5.2"},{"comment":"The second-rater reliability check was performed on the ten-database judging round and not on the eleven-database round reported in the main text; this is disclosed, but the main-text presentation should state the limitation before the kappa value is quoted, not only in the subsequent paragraph.","section":"5.3"},{"comment":"The gate-ON no-evidence coverage of 0.000 is described as 'by construction,' which is correct; however, the phrase 'recall-when-claimed never degrades' in the same subsection could be misread as a general empirical guarantee when the measured differences are flat on Gemini and held-out data and positive only on the two speculating backbones.","section":"5.2.8"}],"recommendation":"major_revision","confidential_remarks":"The paper is methodologically strong and unusually honest, and the registered-prediction discipline is a model for the field. However, the training-data contamination concern is genuine and will be raised by any knowledgeable reviewer; the manuscript should address it head-on before publication. The mismatch between the central mechanism claim and the backbone-swap/ablation results also needs to be resolved in the text, even if the resolution is a more modest claim. I would be comfortable with acceptance after a revision that adds a non-public or freshly generated contamination control and reframes the abstraction and contribution statements to match the measured mechanism."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about arXiv:2608.07946. First, its headline result is about selection, not prose: the Rosetta harness doesn't write better column metadata than the same LLM used directly — on the 283 columns where both speak it's measurably worse on two of three facets — its entire gain is declining when deterministic evidence is weak (coverage 0.550 with structural evidence vs 0.293 without, +0.257, interval excluding zero on Gemini). Second, this is one of the more honestly measured systems papers I've read: registered predictions with commit hashes, a frozen experimental record, cluster bootstraps over databases, external ground truth from BIRD and i2b2, and three separate analyses showing its own authority ladder is not the mechanism behind the result. The backbone swap is the transfer test most papers skip, and it pays off: the prose finding reproduces almost exactly on Claude Sonnet, but coverage nearly doubles and the evidence-tracking gap disappears.\n\nThe reproducibility infrastructure earns its keep: a frozen experimental record that regenerates every number from committed data, six predictions registered before the commit gate was implemented, and cost transparency ($281 for the whole program). The paper's own negative results — the authority ladder contributes nothing to routing, the checksum-prevalence prediction missed by two orders of magnitude, cold reconstruction doesn't out-generate naive on BIRD — are reported at the same resolution as the positives.\n\nThe soft spots are real but scoped. The abstract overstates the ICD-9 result: it says Rosetta 'decodes 95.5% of 134 real ICD-9 codes from values alone,' but Section 5.4.1 reports the harness adds nothing there — the plain model also scores 0.745 and also abstains on all 44 NDC codes. That discrimination is backbone behavior, and the abstract needs to say so. Relatedly, the memorization concern is legitimate for the decode task: the value 493.90 carries no information about asthma, so the mapping has to come from training memory, and the paper's contamination caveat appears only in the join-graph section, not for the headline reconstruction or the ICD-9 numbers. It's the right question to press on. That said, the concern doesn't sink the central claim. The headline comparison is harness versus same-model-direct on identical inputs, so memorization would inflate both arms, and the relative selection advantage likely survives. For the column-description task specifically, arm A collapsing from 0.752 with names to 0.074 without them weakens the documentation-memorization hypothesis. A contamination-controlled run on proprietary or freshly generated warehouses would settle it.\n\nWho this is for: anyone working on text-to-SQL reliability, schema linking, or data profiling; the selection-over-prose finding and the backbone-transfer data are the takeaways. This deserves a serious referee. I'd accept it conditioned on an abstract rewrite that attributes ICD-9 decoding to the language model rather than the harness, plus a contamination control on the headline numbers.","headline":"A scrupulously measured system paper: the selection-over-prose finding holds, but the abstract over-credits the harness on ICD-9 decoding and the 'values alone' framing is vulnerable to a memorization confound that needs a contamination control.","tokens_in":51250,"tokens_out":10078,"would_cite":true,"duration_ms":95957,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A verification harness can recover column semantics from values alone, because it declines to guess without grounded evidence.","keywords":["metadata reconstruction","text-to-SQL","abstention","selective prediction","column semantics","data profiling","verification harness","schema opacity"],"falsifier":"Run the reconstruction comparison on a private, never-published warehouse, or on schemas constructed after the model's training cutoff, with identifiers destroyed and documentation withheld; if the harness no longer beats the direct model on recall-when-claimed, or if coverage no longer tracks deterministic evidence, the 'from values alone' claim is falsified.","tokens_in":49997,"feed_emoji":"🗄️","tokens_out":4231,"duration_ms":47611,"temperature":0.7,"pith_summary":"The paper attacks the regime where text-to-SQL benchmarks do not: production warehouses with cryptic column identifiers and missing documentation. It claims that a column's meaning can be recovered from its values alone, provided a language model is placed inside a verification harness: a deterministic profiler decides whether the system speaks, and the model writes only when the evidence licenses it. Measured against human-authored documentation, the harness recovers more of what authors wrote (0.475 accurate on the 42% of columns it commits to, against 0.223 on 94% for the same model used directly), but the paper shows the gain is selection, not better prose. On the reconstructed catalog, the system answers selectively with calibrated abstention, converting a naive model's confident silent errors into refusals. If right, this reframes the bottleneck in querying undocumented warehouses as metadata quality rather than translation quality.","feed_headline":"Column meanings recovered from values alone—when the model is gated","feed_subtitle":"On stripped schemas, the harness is accurate on 42% of columns vs 23% for the model alone—by declining to speak when evidence is thin.","key_machinery":"The central object is the verification harness: a deterministic profiler extracts structural evidence (value fingerprints, a 26-pattern library, checksum verdicts), a language model proposes semantics conditioned on that evidence, and every fact carries provenance with a confidence capped by its evidence class. A code-enforced commit gate refuses prose claims whose grounding tier is too weak, and at query time the route (answer / confirm-first / refuse) is pure thresholded arithmetic over a feature vector in which the LLM contributes features but never decides the route.","core_discovery":"The paper claims that when a column is stripped of its name and documentation, the Rosetta architecture recovers more of what human authors wrote than either a statistical semantic-type detector or the same language model used directly, because it is the only one of the three that declines when it cannot ground an answer, and because that declining is driven by deterministic evidence rather than by the model's self-report. The paper is explicit that the advantage is selection, not prose: on the 283 columns where both arms speak, the harness writes no better metadata than the model alone, and on two facets measurably worse. Separately, over the reconstructed catalog, the system delivers calibrated abstention at query time, converting a naive model's confident silent errors into auditable refusals, with coverage that tracks difficulty. The paper also reports, at length, that its own authority ladder is not the mechanism behind the abstention; execution grounding and the deterministic threshold are.","pith_inferences":["If selection is the real gain, cheaper deterministic competence detectors might capture much of the benefit without an LLM; the paper's own evidence-split on BIRD columns suggests a testable extension of this kind.","The code-enforced commit gate result implies that any system claiming selective answering should register its refusals in code, not in prompt language, because model disposition varies across backbones.","A testable extension would apply the same value-decoding protocol to proprietary, registry-assigned code systems, where no world knowledge exists; the paper's NDC result predicts the gate abstains on all such keys.","The finding that execution grounding carries the routing signal suggests that query-time safety may be improved more by execution verification and result-sanity checks than by richer metadata confidence scores."],"forward_implications":["Undocumented warehouses can be bootstrapped into query-answerable catalogs without human schema curation, with each reconstructed fact carrying an audit trail.","Deploying a naive text-to-SQL system on an opaque schema should be expected to fail silently on a majority of questions; an evidence-gated pipeline converts those silent errors into refusals.","The measurable value of a language-model harness in this setting is selecting which columns to describe, not writing better descriptions.","Abstention must be enforced in code rather than requested in a prompt, since prompt-requested abstention transfers poorly across model families.","A catalog that is 0.475 accurate on 42% of columns is more operationally useful than one that is 0.223 accurate on 94%, because the second kind cannot be trusted anywhere."],"supporting_citations":[{"why":"Supplies the public benchmark whose human-authored column documentation is the ground truth for the reconstruction measurements.","marker":"[16]"},{"why":"Supplies the cross-domain text-to-SQL benchmark and execution-labeled questions used for the query-time calibrated-abstention study.","marker":"[28]"},{"why":"Defines the closest abstention-aware text-to-SQL system against which Rosetta positions its authority-bounded reconstruction approach.","marker":"[2]"},{"why":"Provides the multivariate Platt-scaling calibration baseline that the router baseline matrix is compared against.","marker":"[17]"},{"why":"Surveys correctness signals for selective prediction in text-to-SQL, defining the comparison set for the router evaluation.","marker":"[23]"},{"why":"Represents the learned semantic-type detection tradition that the statistical arm stands for in the three-way reconstruction comparison.","marker":"[13]"},{"why":"Supplies the inclusion-dependency discovery technique used by the data-scan verifier for foreign-key reconstruction.","marker":"[20]"}],"fun_headline_variants":["Gated model recovers column meanings from values alone","Declining on thin evidence boosts metadata recovery","Deterministic abstention gate, not prose, powers schema recovery","Calibrated abstention: from silent errors to audited refusals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline results are measured on public benchmark data that a frontier language model may have memorized, so the claim that meanings are recovered 'from values alone' depends on the model not recalling the documentation or ontology mappings instead of deriving them from the values.","fun_headline_variants_meta":{"raw":{"variants":["Gated model recovers column meanings from values alone","Declining on thin evidence boosts metadata recovery","Deterministic abstention gate, not prose, powers schema recovery","Calibrated abstention: from silent errors to audited refusals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000317,"raw_usage":{"total_tokens":1879,"prompt_tokens":1114,"completion_tokens":765,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":730,"completion_tokens_details":{"reasoning_tokens":696}},"tokens_in":730,"tokens_out":765,"duration_ms":13232,"temperature":1.0,"reasoning_tokens":696,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:38:31.933069+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the reconstruction comparison on a private, never-published warehouse, or on schemas constructed after the model's training cutoff, with identifiers destroyed and documentation withheld; if the harness no longer beats the direct model on recall-when-claimed, or if coverage no longer tracks deterministic evidence, the 'from values alone' claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the multivariate Platt-scaling calibration baseline that the router baseline matrix is compared against."},{"cited_title":"What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study","cited_arxiv_id":"2607.06799","evidence_quote":"Surveys correctness signals for selective prediction in text-to-SQL, defining the comparison set for the router evaluation."},{"cited_title":"Sherlock: A Deep Learning Approach to Semantic Data Type Detection","cited_arxiv_id":"1905.10688","evidence_quote":"Represents the learned semantic-type detection tradition that the statistical arm stands for in the three-way reconstruction comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the inclusion-dependency discovery technique used by the data-scan verifier for foreign-key reconstruction."}],"review_version":1}