{"id":"aaf0f5e3-1af9-4889-af89-b8702255fa0c","arxiv_id":"2611.17044","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLMs recover table structure from natural language but consistently under-perform on foreign keys, data types, and nullability, while detailed prompting sharply improves round-trip schema preservation despite lowering BLEU, ROUGE-L, and BERTScore.","lead":"This paper introduces SchemaBridge, a benchmark of 902 database schemata that tests whether large language models can design schemata from natural language, describe existing schemata, and preserve structure through a design-describe-redesign round trip. It finds LLMs recover tables well but struggle with referential and domain constraints, and that asking for detailed descriptions sharply improves round-trip fidelity even as text-similarity scores drop.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The claim that human-written requirements under-specify schemata rests on unverified authorship and an untested interpretation of low F1; the paper's own §IV-A disclosure (b147) undermines the 'human-written' qualifier.","rationale":"I confirm the reader's CONDITIONAL verdict. The most load-bearing concern is the interpretation attached to finding (i), not the measurements themselves. The abstract asserts 'human-written requirements under-specify schemata' (b4), while Section IV-A discloses that RSchema requirements were partially LLM-generated and that SchemaBench's documentation does not establish human authorship (b147); the paper's own text thus undermines the 'human-written' qualifier of its headline claim, and the conclusion (b285) repeats the claim unhedged. This is an internal tension, not a disagreement with external consensus. Second, even granting authorship, the inference from 'no model exceeds 0.27 referential F1' to 'requirements under-specify' presupposes that a human designer given the same requirements would match the gold schema; that premise is untested. Because matching is strict reference-based set matching (§III-E) and referential integrity requires exact 4-tuple agreement, valid alternative designs are penalized — a confound the paper itself acknowledges (b173; Introduction). The Table VI baselines, where few-shot prompting raises referential F1 3–4x, show the scores are sensitive to extraction behavior, complicating the under-specification reading. Credit where due: the round-trip finding (ii) is a paired comparison against the same gold DDL and survives these concerns, as do the masking retention results (iv); the disclosures at b147, b173, and b255–257 are honest and specific. The proposed human-expert baseline plus authorship verification would settle whether finding (i) describes requirement content or LLM performance under strict scoring. Since the framework, round-trip diagnosis, and masking findings retain value either way, CONDITIONAL remains the right verdict; at stake is the wording and interpretation of finding (i), not the benchmark itself.","tokens_in":20404,"tokens_out":15216,"duration_ms":115393,"concrete_test":"Run a human-expert baseline on the same requirements with the identical SchemaBridge scoring (§III-E): recruit 3–5 independent database designers to produce DDL for a stratified sample (≥30 per dataset) of RSchema and SchemaBench requirements, given only the requirement text and the constructor prompt's output specification. Compare their micro-averaged referential-integrity F1 with the 0.27 LLM ceiling. If expert human F1 stays near or below 0.27, the requirements genuinely under-specify and finding (i) survives (modulo authorship).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise for finding (i) — 'human-written requirements under-specify schemata, with no model exceeding 0.27 referential-integrity F1' (abstract, b4) — is threefold: (a) the RSchema and SchemaBench requirement texts are genuinely human-authored and typical; (b) the gold schemata are correct; (c) the low F1 reflects missing information in the requirements rather than the strictness of exact-match scoring or constructor failure. Premise (a) is contradicted for RSchema, whose requirements the paper says were partially LLM-generated (Section IV-A), and unverified for SchemaBench, whose documentation 'does not... indicate whether the specifications were written entirely by humans or with LLM assistance' (b147); the abstract and conclusion (b285) assert the claim unhedged. Premise (c) is untested: matching is strict reference-based set matching (§III-E), the referential dimension requires an exact 4-tuple match (the harshest criterion), and the paper itself concedes that exact matching counts valid alternative designs as errors (b173; Introduction). No human control establishes how much of the 0.27 ceiling is mechanical. The paper's own baselines (Table VI) show few-shot prompting moving referential F1 on RSchema from 0.004–0.146 to 0.118–0.188, a 3–4x swing, indicating the scores are sensitive to extraction behavior rather than purely to information content. Findings (ii)–(iv) are paired comparisons against the same gold DDL and survive this concern; finding (i), the abstract's first headline, is unsupported as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper presents SchemaBridge, a benchmark for evaluating LLMs on three relational schema reasoning tasks: Text2Schema (design from NL), Schema2Text (verbalization), and Schema→Text→Schema (round-trip preservation). It uses 902 schemata from three collections, stratified by size and constraint complexity, and measures structural fidelity across five dimensions (tables, attributes, domain, entity, referential integrity). Six LLMs are evaluated. Key findings: (i) human-written requirements under-specify schemata, with low referential F1; (ii) detailed prompting improves round-trip fidelity; (iii) constructor and describer choices matter; (iv) structural cues aid recovery of keys/relationships but domain info is poorly retained.","tokens_in":20792,"tokens_out":1307,"duration_ms":12820,"significance":"If the findings hold, SchemaBridge provides a much-needed unified evaluation framework for schema-level reasoning, moving beyond Text2SQL. The per-dimension fidelity metrics and round-trip diagnosis are methodologically sound and practically useful. The finding that text-similarity metrics do not reflect structural preservation is important for the field. The benchmark and code are released, supporting reproducibility.","major_comments":[{"comment":"The claim that 'human-written requirements under-specify schemata' (abstract, b4; §V-B) is load-bearing but rests on unverified premises about the datasets. As the paper itself discloses (§IV-A, b147), RSchema requirements were partially LLM-generated, and SchemaBench's documentation does not confirm human authorship. The abstract and conclusion assert the claim unhedged. The paper should either (a) provide evidence that the requirements are representative of human-authored specifications, or (b) soften the claim to 'the evaluated requirements (which may include LLM-generated content) under-specify schemata relative to the gold references.' Without this, finding (i) is ambiguous.","section":"Abstract, §V-B, §IV-A","authors_acknowledge":"The dataset documentation states that the examples were manually curated and semi-automatically validated, but it does not describe the validation procedure or indicate whether the specifications were written entirely by humans or with LLM assistance.","authors_acknowledge_p":0.6},{"comment":"The low referential F1 (≤0.27) is interpreted as requirements under-specifying schemata, but the paper does not control for the strictness of exact-match scoring. Referential integrity requires an exact 4-tuple match (§III-E), which is the harshest criterion. The paper concedes that exact matching counts valid alternative designs as errors (b173). Without a human baseline or an alternative scoring that rewards partial matches, the extent to which the low F1 reflects missing information vs. scoring strictness is unknown. The authors should discuss this limitation explicitly and consider reporting a relaxed metric (e.g., allowing equivalent foreign keys with different column orderings or naming).","section":"§V-B, Table V, §III-E","authors_acknowledge":"Note that EASYdoes not imply easier reconstruction: constraint-sparse schemata leave an LLM more room to add structure that the gold design omits, which exact matching counts as errors (§V-B and V-F).","authors_acknowledge_p":0.75},{"comment":"The baseline comparison (Table VI) shows that few-shot prompting improves referential F1 from 0.004–0.146 to 0.118–0.188, a 3–4x swing. This indicates that the scores are sensitive to extraction behavior (prompting strategy) rather than purely to information content. The paper acknowledges this indirectly but does not reconcile it with the 'under-specify' interpretation. If a simple few-shot prompt can triple referential F1, the ceiling may be higher than 0.27 under optimal prompting, undermining the claim that requirements inherently lack information. The authors should discuss this.","section":"§V-B, Table VI","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The normalization of identifiers (lowercasing, removing underscores, etc.) is reasonable but may conflate distinct identifiers (e.g., 'order_id' vs 'orderid'). The authors should note this limitation.","section":"§III-B","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The stratification thresholds (5/15 tables, 50/150 columns, s=4/10) are described as heuristic. The paper should justify these thresholds more concretely or show that results are robust to small variations.","section":"§IV-B, Table III","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"SCR is reported as macro-average, but the paper does not specify whether it is macro- or micro-averaged. Clarify.","section":"§V-C, Table VII","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The round-trip results for RSchema show very high F1 (e.g., 0.994 for tables under detailed prompting). This is surprising given the low Text2Schema scores. The authors should comment on why round-trip preservation is so high for RSchema despite its requirements being underspecified.","section":"§V-D, Table VIII","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The heuristic inferability labels are described as reflecting structural clues rather than logical entailment. The paper should provide examples of what makes an element 'inferable' vs 'non-inferable' to aid reproducibility.","section":"§V-F, Table X","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The cost analysis uses only 30 schemas per dataset. The authors should note that these estimates may not generalize to the full dataset.","section":"§V-G, Table XII","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The recommendations are sensible but some are generic (e.g., 'Specify structural requirements explicitly'). The authors could tie them more directly to specific findings.","section":"§VI","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper uses 'human-written requirements' in the abstract and conclusion but the datasets include LLM-generated content. This should be corrected for accuracy.","section":"General","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The paper builds SchemaBridge, a benchmark that jointly evaluates Text2Schema, Schema2Text, and Schema→Text→Schema round-trip preservation across 902 schemata, with per-dimension structural fidelity scoring (tables, attributes, domain, entity, referential integrity). This unified treatment is genuinely new relative to SchemaAgent, SchemaBench, and Avignone et al., which each cover only part of the space. The methodology is careful: identifier and type normalization, micro-averaging, failures treated as empty predictions, and disclosed restrictions on domain evaluation. The round-trip and masking protocols are well-designed and yield internally consistent measurements. The finding that detailed prompting dramatically improves round-trip referential F1 (0.228→0.909 for GPT-5.4-mini on SchemaBench) while text-similarity metrics decrease is a useful empirical result. The paper also ships code and data, which is real evidence.\n\nThe soft spot is the headline claim that \"human-written requirements under-specify schemata\" (abstract, b4). The paper itself discloses (b147) that RSchema requirements were partially LLM-generated and SchemaBench's documentation does not indicate whether specifications were written entirely by humans or with LLM assistance. So the qualifier \"human-written\" is unverified for both datasets used in the Text2Schema evaluation. The stress-test note is correct on this point. The paper also concedes that exact matching counts valid alternative designs as errors (b173), and the referential dimension requires an exact 4-tuple match, which is the harshest criterion. No human control establishes how much of the 0.27 ceiling is mechanical. The paper's own baselines (Table VI) show few-shot prompting moving referential F1 on RSchema from 0.004–0.146 to 0.118–0.188, a 3–4x swing, indicating the scores are sensitive to extraction behavior rather than purely to information content. So finding (i) is unsupported as stated. Findings (ii)–(iv) are paired comparisons against the same gold DDL and survive this concern.\n\nWho this is for: researchers working on LLM-based schema generation, schema verbalization, and benchmark design. The benchmark itself is reusable and the per-dimension scoring is a methodological contribution. The paper deserves a serious referee, but the authors should be asked to either verify the provenance of the requirement texts or soften the claim about human-written requirements. I would cite the benchmark methodology but not the headline claim without verification.\n\nRecommendation: Accept for peer review with a request to address the data provenance issue for the Text2Schema findings.","headline":"SchemaBridge is a careful, reusable benchmark for LLM schema reasoning, but its headline claim about human-written requirements is weaker than advertised because the data provenance is unverified.","tokens_in":21382,"tokens_out":637,"duration_ms":6783},"pith_extraction":{"msc":["68P15","68T50"],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"LLMs can preserve relational schemata through language if prompted thoroughly, but cannot design them from human requirements.","keywords":["relational schema","schema design","schema comprehension","large language models","Text2Schema","Schema2Text","round-trip preservation","structural fidelity"],"falsifier":"If a new dataset of verified human-authored requirements with validated gold schemata showed any LLM achieving referential integrity F1 above 0.5 in direct Text2Schema, or if a round-trip experiment with a different describer prompt showed domain retention above 0.3, the paper's central claims about underspecification and domain fragility would be weakened.","tokens_in":21813,"feed_emoji":"🗄️","tokens_out":1556,"duration_ms":13067,"temperature":0.7,"pith_summary":"This paper argues that large language models are poor at constructing a relational database schema from a natural-language requirement (Text2Schema), with no tested model exceeding 0.27 F1 for referential integrity. However, when an existing schema is described in natural language with an explicit instruction to be exhaustive, a later reconstruction from that description can recover most of the original structure: GPT-5.4-mini's referential F1 on SchemaBench rises from 0.228 to 0.909. The authors introduce SchemaBridge, a benchmark of 902 schemata from three collections, and measure fidelity separately for tables, attributes, domain (data types and nullability), entity integrity, and referential integrity. They also show that text-similarity metrics like BLEU and ROUGE-L do not track structural preservation, that the choice of describer and constructor both matter independently, and that data-type and nullability information is especially fragile through verbalization.","feed_headline":"LLMs preserve schemata through language but can't design them","feed_subtitle":"With detailed prompts, GPT-5.4-mini recovers 91% of foreign keys from its own description — but direct schema design tops out at 27%.","key_machinery":"SchemaBridge is the central object: a unified benchmark of 902 relational schemata from Enterprise-145, RSchema, and SchemaBench, stratified by size and constraint complexity. It defines structural fidelity as exact set matching after normalization across five dimensions: tables, attributes, domain integrity (data type + nullability), entity integrity (primary keys and unique constraints), and referential integrity (foreign keys). The round-trip protocol (Schema→Text→Schema) with controlled describer and constructor isolation is the mechanism that separates verbalization loss from reconstruction loss.","core_discovery":"The paper establishes that LLMs can preserve relational schema structure through a Schema→Text→Schema round trip when the describer is prompted to be exhaustive, even though they cannot reliably construct a schema from typical human-authored requirements. The key empirical finding is that GPT-5.4-mini's referential integrity F1 on SchemaBench jumps from 0.228 (direct Text2Schema) to 0.909 (round trip with detailed description), while BLEU-4, ROUGE-L, and BERTScore all decrease. This reveals that standard text-similarity metrics are misleading for evaluating schema comprehension, and that the bottleneck is not the LLM's ability to reason about relational structure but the underspecification of natural-language requirements.","pith_inferences":["The paper's results imply that a practical schema-documentation pipeline could use an LLM describer with a detailed prompt followed by automated validation of types and nullability, rather than trying to improve Text2Schema directly.","The finding that few-shot prompting improves Text2Schema over direct prompting (Table VI) while one-shot CoT hurts it suggests that the models rely on surface pattern matching from examples rather than genuine reasoning about relational design.","The poor retention of domain information (7.8%) combined with the strong recovery of keys and foreign keys when structural cues are present suggests that LLMs treat data types as arbitrary labels rather than semantically meaningful constraints.","If the RSchema requirements are partially LLM-generated (as the paper notes is unverified), then the \"human-written requirements under-specify schemata\" conclusion may be an artifact of the dataset rather than a property of human requirements."],"forward_implications":["Schema design from natural language should not be trusted without manual validation; current LLMs miss most referential constraints even with prompting.","For schema documentation or interchange, a detailed prompting strategy that explicitly requests tables, columns, types, keys, and foreign keys can yield descriptions from which the original schema is largely recoverable.","Text-similarity metrics (BLEU, ROUGE-L, BERTScore) are inadequate for evaluating schema descriptions; structural reconstruction metrics should replace them in any serious evaluation.","Data types and nullability are the most fragile schema dimension through verbalization, with only 7.8% retention from direct to round-trip recall, suggesting that domain information needs special handling.","The describer model and the constructor model should be selected independently; a model that describes well is not necessarily a good reconstructor."],"supporting_citations":[{"why":"SchemaAgent provides the closest prior Text2Schema work and supplies the RSchema dataset used in the benchmark.","marker":"[11]"},{"why":"SchemaBench provides one of the three schema collections (239 schemata with natural-language specifications).","marker":"[21]"},{"why":"Enterprise-145 is derived from the CTU Prague Relational Learning Repository, providing the third schema collection.","marker":"[29]"},{"why":"CodeLlama-13B is one of the six LLMs evaluated as both describer and constructor.","marker":"[30]"},{"why":"DeepSeek-Coder-6.7B is one of the six LLMs evaluated.","marker":"[31]"},{"why":"Llama-3.1-8B is one of the six LLMs evaluated.","marker":"[32]"},{"why":"Claude Sonnet 4.6 is one of the proprietary LLMs evaluated.","marker":"[33]"},{"why":"GPT-5.4-mini is the proprietary model that achieves the key round-trip result (referential F1 0.909).","marker":"[34]"},{"why":"GPT-4o-mini serves as the fixed constructor for round-trip and masking experiments.","marker":"[35]"}],"fun_headline_variants":["LLMs talk schemas but can't write them from scratch","Schemas survive a round trip through text—but only if asked thoroughly","GPT-5.4-mini recovers 91% of foreign keys in a Schema→Text→Schema loop","Text-to-schema fails; Schema-to-text-to-schema nearly works","Why LLMs can't design database schemas from plain English"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The claim that human-written requirements under-specify schemata depends on the premise that the requirements in RSchema and SchemaBench are genuinely human-authored and that their gold schemata are correct references, but the paper discloses that RSchema requirements were partially LLM-generated and SchemaBench's validation procedure is undocumented.","fun_headline_variants_meta":{"raw":{"variants":["LLMs talk schemas but can't write them from scratch","Schemas survive a round trip through text—but only if asked thoroughly","GPT-5.4-mini recovers 91% of foreign keys in a Schema→Text→Schema loop","Text-to-schema fails; Schema-to-text-to-schema nearly works","Why LLMs can't design database schemas from plain English"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":9.6e-05,"raw_usage":{"cost":9.63872e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":9.63872e-05,"upstream_inference_prompt_cost":7.83692e-05,"upstream_inference_completions_cost":1.8018e-05},"total_tokens":1165,"prompt_tokens":1066,"completion_tokens":99,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":0}},"tokens_in":1066,"tokens_out":99,"cost_source":"openrouter_usage","duration_ms":4223,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":0,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T07:14:13.507462+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"If a new dataset of verified human-authored requirements with validated gold schemata showed any LLM achieving referential integrity F1 above 0.5 in direct Text2Schema, or if a round-trip experiment with a different describer prompt showed domain retention above 0.3, the paper's central claims about underspecification and domain fragility would be weakened.","supporting_citations":[{"cited_title":"SchemaAgent: A multi-agent framework for generating relational database schemas,","cited_arxiv_id":null,"evidence_quote":"SchemaAgent provides the closest prior Text2Schema work and supplies the RSchema dataset used in the benchmark."},{"cited_title":"SchemaBench,","cited_arxiv_id":null,"evidence_quote":"SchemaBench provides one of the three schema collections (239 schemata with natural-language specifications)."},{"cited_title":"The ctu prague relational learning repository,","cited_arxiv_id":null,"evidence_quote":"Enterprise-145 is derived from the CTU Prague Relational Learning Repository, providing the third schema collection."},{"cited_title":"Code Llama: Open Foundation Models for Code","cited_arxiv_id":"2308.12950","evidence_quote":"CodeLlama-13B is one of the six LLMs evaluated as both describer and constructor."},{"cited_title":"DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence","cited_arxiv_id":"2401.14196","evidence_quote":"DeepSeek-Coder-6.7B is one of the six LLMs evaluated."},{"cited_title":"The Llama 3 Herd of Models","cited_arxiv_id":"2407.21783","evidence_quote":"Llama-3.1-8B is one of the six LLMs evaluated."},{"cited_title":"Introducing claude sonnet 4.6,","cited_arxiv_id":null,"evidence_quote":"Claude Sonnet 4.6 is one of the proprietary LLMs evaluated."},{"cited_title":"Introducing GPT-5.4 mini and nano,","cited_arxiv_id":null,"evidence_quote":"GPT-5.4-mini is the proprietary model that achieves the key round-trip result (referential F1 0.909)."},{"cited_title":"GPT-4o mini: Advancing cost-efficient intelligence,","cited_arxiv_id":null,"evidence_quote":"GPT-4o-mini serves as the fixed constructor for round-trip and masking experiments."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}