{"id":"df8b7cc6-f6bf-44da-bbc8-cfaf01791afb","arxiv_id":"2608.08459","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Doc2DB-Bench provides 203 long-document, database pairs to test whether LLMs can reconstruct multi-table relational databases with correct keys, relationships, and constraints.","lead":"This paper introduces Doc2DB-Bench, a benchmark that tests whether AI language models can turn long documents into relational databases instead of just flat tables. It matters because finance, healthcare, and enterprise systems need reliable extraction of entities, relationships, and constraints from reports for analytics and compliance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth integrity rests only on unverified synthesis validation: Definition 2 completeness/exclusiveness is checked by the same LLM family that generated the documents, so Table 3 scores are conditional on an unaudited assumption.","rationale":"The reader's conditional verdict is already built on the right worry: the benchmark's ground truth is only as good as the synthesis pipeline's completeness/exclusiveness guarantee. I agree that this is the load-bearing assumption. Without an independent audit, the benchmark could systematically reward or penalize models for reasons unrelated to document-to-database ability. My concern is not an internal inconsistency in the definitions; Definition 2 is clear, and the pipeline attempts to satisfy it. The problem is that the operational checks in Step 5 cannot certify exclusiveness. Coverage verification checks only that gold evidence is present, and extraction consistency checks recoverability of D*, not the absence of extra supporting facts. The human verification is described at a high level with no measurement of completeness or exclusiveness, and the paper's own Limitations section concedes that human verification is important. The unrelated inserted DataMosaic block is a real submission defect and should be cleaned up, but I do not rest the objection on it; the ground-truth audit is more central. The proposed test — a blind, tag-stripped fact-exhaustion audit on a sample of instances — directly tests Definition 2 and would either validate the benchmark or identify the affected instances. Because this audit is missing, the current conditional verdict is appropriate; the concern does not, by itself, justify rejection, since an independent audit could settle it favorably.","tokens_in":26705,"tokens_out":4351,"duration_ms":50043,"concrete_test":"Stratified fact-exhaustion audit: sample 30 Doc2DB-Bench instances covering all seven domains, strip the provenance tags from the synthesized documents, and give each tag-stripped document plus its schema to two independent annotators blind to the gold database. Ask each annotator to enumerate every entity, attribute value, and relationship they can infer from the document. Then compare each annotator's fact set against the gold database D*. If any annotator derives a fact not in D* or cannot derive a D* fact from the document, Definition 2 fails for that instance; the affected instances should be repaired or excluded, and Table 3 should be recomputed on the repaired benchmark.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Doc2DB-Bench's central claim — that the Table 3 model scores measure relationally faithful document-to-database construction — requires Definition 2's completeness and exclusiveness to hold for all 203 instances. The paper's only direct evidence is Step 5's dual validation: coverage verification via provenance tags (which checks that gold atoms appear, not that unsupported facts are absent) and extraction consistency (an LLM re-extracts D'_syn from D_syn and compares it to D*). Neither check can establish exclusiveness. The provenance tags force the writer to include gold facts, but nothing forces the writer to avoid adding extra facts; the extractor is in the same Gemini-2.5-Pro family as the generator, so a spurious statement that the extractor fails to recover or that is phrased as non-committal is invisible to the check. Section 5.4's authenticity rubrics and AI-detector scores assess style and genre fidelity, not fact-set exclusiveness. Appendix A.4 says seven PhD candidates reviewed documents, but no protocol, inter-annotator agreement, or blind fact-exhaustion audit is reported; the Limitations section explicitly concedes that human verification is important for ensuring quality, which implies the automated checks alone were not considered sufficient. If any document omits evidence or entails facts beyond D*, the gold database for that instance is corrupted, making that instance's precision, recall, and F1 scores meaningless. This is the single most load-bearing assumption, and it is not independently audited in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Doc2DB-Bench, a benchmark for evaluating LLMs on constructing normalized relational databases from long documents, rather than extracting flat tables. The benchmark comprises 203 document instances built from 42 BIRD/Spider database schemas via a five-stage controllable DB2Doc reverse-synthesis pipeline, with a two-pillar capability taxonomy covering intra-table and inter-table reasoning. The authors report fine-grained capability annotations, evaluate 14 open-source, proprietary, and specialized IE systems on entity-level, relation-level, and overall F1 scores, and claim that the generated documents are authentic and indistinguishable from real-world references. The contribution is positioned as the first benchmark for systematic LLM evaluation on document-to-database construction.","tokens_in":26960,"tokens_out":5411,"duration_ms":57137,"significance":"If the central validity assumption holds, Doc2DB-Bench fills a genuine gap: existing benchmarks such as DTBench and SQUiD do not test normalized multi-table construction from long heterogeneous documents with entity identity, cross-table relationships, and integrity constraints. A notable strength is that the gold databases come from external BIRD and Spider sources, so the core target labels are not derived from the same synthesis loop that generates the documents. The capability taxonomy and fine-grained annotations are also useful for diagnosing model failures, and the broad baseline comparison across fourteen systems is valuable. However, the significance is contingent on two load-bearing requirements: that the synthesized documents satisfy the stated completeness and exclusiveness conditions with respect to the ground-truth database, and that the authenticity and reproducibility claims are supported by the released artifacts.","major_comments":[{"comment":"The exclusiveness condition of Definition 2 is load-bearing for every score in Table 3, but the validation described in Step 5 cannot establish it. Coverage verification checks only that tagged atomic evidence is present, not that unsupported facts are absent; extraction consistency re-extracts a database from the generated document using a Gemini-2.5-Pro validator, which is the same model family as the generator and is not an exhaustive entailment check. Appendix A.4 mentions human review by seven PhD candidates, but reports no protocol, no blind fact-exhaustion audit, and no inter-annotator agreement. If any document entails facts beyond the ground-truth database, the gold instance is corrupted and that instance's precision, recall, and F1 are uninterpretable. Please provide an independent audit of exclusiveness (for example, adversarial human fact-mining with agreement statistics) or a formal argument that the generation protocol prevents extra facts.","section":"Definition 2 and Section 4, Step 5"},{"comment":"The abstract claims the generated documents are 'proving indistinguishable from real-world references,' but Table 5 reports ZeroGPT scores of 35.20 for Doc2DB-Bench documents versus 23.12 for reference documents. The synthesized documents are therefore flagged as AI-generated substantially more often than the references, directly contradicting the stated claim. The FastGPT comparison is closer, but the ZeroGPT gap of about 12 points should be explained or the claim should be weakened.","section":"Abstract and Table 5"},{"comment":"GPT-4o is both a baseline system in Table 3 and the LLM judge used for semantic quality evaluation in Section 5.1; the paper reports no analysis of evaluator bias, no agreement between the LLM judge and human ratings, and no ablation with alternative judges. The Step 5 extraction-consistency validator also belongs to the same Gemini-2.5-Pro family as the document generator. This shared-model-family circularity does not by itself invalidate the entity/relation scores, because those are computed against the external gold databases, but it does affect the semantic-quality scores and the authenticity evaluation. Please report judge agreement and assess whether the shared model family can inflate any reported number.","section":"Section 5.1 and Table 3"},{"comment":"The reproducibility of the benchmark is not yet demonstrated. The paper states the benchmark is publicly available at a GitHub URL, but it provides no commit hash, no dataset version, no code for the synthesis pipeline, no exact configuration files, and no documented procedure for reproducing the human verification. Without these artifacts, the claimed controllability of the pipeline and the exact numbers in Section 5 cannot be independently checked. Please include a complete artifact appendix or a permanent DOI with datasets, prompts, and run instructions.","section":"Section 1 and Appendix A"},{"comment":"The manuscript contains a large unintegrated block from a different paper, 'Document-to-Database: Extraction Meets Relational Semantics,' in PVLDB format, beginning shortly after Section 1 and including its own abstract, figures, and definitions. This material is not part of the Doc2DB-Bench paper and is not referenced or integrated. As submitted, the paper is internally incoherent and cannot be reviewed as a single work; the block must be removed or clearly separated from the main text.","section":"Manuscript structure, after Section 1"}],"minor_comments":[{"comment":"The text says the benchmark contains '42 synthesized (Document, Specification, Database) samples,' while Table 2 lists 203 documents and 42 databases; the wording should be corrected to avoid the impression that there are only 42 document instances.","section":"Section 4, statistics paragraph"},{"comment":"The number of human evaluators is inconsistent: Section 5.4 reports 'five graduate students,' while Appendix A.4 reports 'seven PhD candidates.' Please align these numbers and describe the exact role of each group.","section":"Section 5.4 and Appendix A.4"},{"comment":"Several taxonomy abbreviations in Figure 2 are not expanded in the caption or immediately surrounding text (for example, O2M, MB, ED, TA, UT, RI, AR). Expanding all abbreviations in the caption would improve readability.","section":"Figure 2"},{"comment":"The notation is inconsistent: Definition 1 uses D for the instantiated database and D* for ground truth, while the DB2Doc synthesis definition uses D* for the given ground truth and D_syn for the synthesized document. Please standardize the notation across Section 2 and Section 4.","section":"Definition 1 and Definition 2"},{"comment":"Some references are incomplete or use placeholder formatting (for example, entries marked 'arXiv:XXXX.XXXXX'), and a few citations in the text are not resolved. A careful reference cleanup is needed.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The embedded block from a prior PVLDB-style paper on DataMosaic should be examined for overlap or duplicate-publication concerns, since it appears to reproduce substantial text and figures from another work by overlapping author groups. The editor may also wish to check whether the benchmark artifacts will actually be released with code and data, because the current GitHub-only pointer is insufficient for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The thing to know: Doc2DB-Bench is a useful idea—evaluating LLMs on constructing relational databases rather than flat tables—and the authors put real work into a controllable synthesis pipeline and a capability taxonomy. But the submitted manuscript is not in reviewable shape. A large block from a different paper (the same group's PVLDB DataMosaic paper) is embedded in the middle of the text, with its own abstract and references. That alone would send me back to the authors.\n\nThe good parts: the gap is real. Existing Doc2Table benchmarks don't test cross-table identity, keys, and integrity constraints. The two-pillar taxonomy (intra-table vs inter-table capabilities) is reasonable, and the fine-grained evaluation is useful. The oracle-entity relation evaluation is a nice way to isolate relational reasoning. The experiments cover a broad range of models, and the database-level metrics (entity coverage, relation coverage, referential integrity) go beyond simple cell matching.\n\nThe soft spots: the ground-truth integrity claim is load-bearing. Definition 2 requires completeness and exclusiveness, but the dual validation in Step 5 does not actually establish exclusiveness. Coverage verification only checks that the gold facts appear; nothing checks that the documents avoid implying extra facts. The extraction-consistency check uses an LLM in the same family as the generator (Gemini-2.5-Pro), so it can miss spurious content that the model itself doesn't recover. Human review is mentioned but no protocol or agreement rates are reported, and the Limitations section implicitly concedes the automated checks aren't sufficient. If any document entails facts beyond the gold database, that instance is corrupted and every score in Table 3 is meaningless. Second, the authenticity claim is undercut by the paper's own Table 5: ZeroGPT gives 35.20 for Doc2DB-Bench vs 23.12 for reference documents—that's a gap, not \"indistinguishable.\" Third, no code or data hash is provided, which is a problem for a benchmark paper.\n\nNet: the benchmark deserves attention, but this submission needs major cleanup and a real audit of the ground-truth assumption (blind fact-exhaustion checks by human annotators, plus code/data release) before I'd want a referee spending time on it. If the authors resubmit with the extraneous text removed, code and data attached, and an independent ground-truth audit, I'd then send it out. As is, I'd return it to the authors.","headline":"Useful benchmark idea with a real gap, but the submitted manuscript is self-sabotaged by an embedded block from another paper and an unaudited ground-truth assumption.","tokens_in":27532,"tokens_out":5525,"would_cite":false,"duration_ms":56543,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces Doc2DB-Bench, the first benchmark for evaluating whether LLMs can reconstruct relational databases—with entity identities, keys, relationships, and integrity constraints—from long documents, and reports that even the…","keywords":["document-to-database extraction","relational database construction","LLM evaluation benchmark","multi-table reasoning","information extraction","synthetic document generation","integrity constraints","long-document understanding"],"falsifier":"Take a random sample of Doc2DB-Bench documents, give them to annotators who have never seen the gold database, and ask them to extract every fact they can. If any annotator infers a fact that is not in the gold database, or fails to recover a fact that is in it, that instance violates exclusiveness or completeness, and the benchmark's ground truth for that instance is unsound.","tokens_in":26482,"feed_emoji":"🗄️","tokens_out":4715,"duration_ms":45153,"temperature":0.7,"pith_summary":"The paper argues that evaluating document understanding as flat table extraction is insufficient: the real need is to test whether a model can reconstruct a normalized relational database, complete with entity identities, keys, cross-table relationships, and integrity constraints, from long heterogeneous documents. To make this testable, it introduces Doc2DB-Bench, the first benchmark for systematic LLM evaluation on document-to-database extraction, containing 203 long-document instances synthesized from real relational database schemas. The benchmark's central claim is that relation-level reasoning—identity resolution, relationship linking, structural inference, and constraint satisfaction—is a distinct skill that current LLMs perform markedly worse than cell-level extraction. If this is right, progress on document-based analytics, compliance, and SQL-backed workflows should be measured at the database level, not at the table level.","feed_headline":"New benchmark shows LLMs falter at relational database construction","feed_subtitle":"A 203-document test finds relation-level extraction, not cell accuracy, is the weak spot.","key_machinery":"The load-bearing mechanism is the controllable DB2Doc reverse-synthesis pipeline: start from a real relational database instance, decompose each cell and each relational tuple into atomic evidence, assign capability labels from the taxonomy, serialize the evidence into a document plan, generate long-form text with configurable style and noise, and run dual validation (coverage verification and extraction consistency) with reject-and-rewrite loops. The completeness and exclusiveness conditions of Definition 2—every gold value is supported by evidence and no additional facts are inferable—are what make the gold database uniquely recoverable, and they therefore make all downstream model scores meaningful.","core_discovery":"The central discovery is a benchmark plus a controllable reverse-synthesis methodology that turns real relational database instances into long documents from which the original database should be uniquely recoverable. Doc2DB-Bench contains 203 instances across 42 schemas and seven domains, with 117 entity tables, 132 relationship tables, 7,341 rows, and 41,935 cells, each annotated along a two-pillar taxonomy of intra-table extraction and inter-table reasoning capabilities. The paper reports that state-of-the-art LLMs achieve entity-level F1 around 80% but relation-level F1 around 65–72%, with structural reasoning and integrity constraints as the persistent failure modes; even the strongest model plateaus near 80% on transitive inference and dynamic change. The authenticity checks—rubric-based human and LLM evaluation plus commercial AI detectors—are offered as evidence that the synthesized documents are indistinguishable from real-world references, supporting the validity of scores obtained on them.","pith_inferences":["If exclusiveness holds only as strongly as the LLM validator's checks, some documents may still support facts beyond the gold database; a cheap audit is to have a different model family independently re-extract a random sample and measure agreement with the gold instance.","The taxonomy could be reused as a training diagnostic: focusing targeted training on conditional linkage and integrity-constraint sub-capabilities might lift relation-level F1 more than generic long-context fine-tuning.","The 42 schemas could be turned into an adversarial generator—perturbing noise level, distractor density, or missing-value patterns—to stress-test constraint satisfaction, an axis the paper's configuration module enables but does not fully explore."],"forward_implications":["Relation-level extraction should become a primary evaluation axis for any document-to-database system; cell-level F1 alone overstates a model's practical capability.","Models that score well on flat-table benchmarks will not automatically succeed on Doc2DB-Bench, and the capability taxonomy will pinpoint which reasoning steps fail.","The synthesis pipeline can generate additional instances from any relational database, enabling scalable benchmark expansion without manual document annotation.","Database-level metrics—entity coverage, relationship coverage, and referential integrity rate—should complement cell accuracy in future extraction evaluation.","Applications such as compliance and analytics should demand schema-conformant, constraint-satisfying outputs, not merely isolated field values."],"supporting_citations":[{"why":"Supplies the source relational databases used as ground truth for synthesis.","marker":"[23]"},{"why":"Supplies additional source schemas and instances for the reverse-synthesis pipeline.","marker":"[45]"},{"why":"The prior single-table document-to-table benchmark that Doc2DB-Bench explicitly extends.","marker":"[51]"},{"why":"The closest prior text-to-relational database task whose single-view limitations motivate the new benchmark.","marker":"[35]"},{"why":"Companion paper that formalizes the Doc2DB problem the benchmark evaluates.","marker":"[47]"},{"why":"The backbone model used for all synthesis agents, making the pipeline reproducible.","marker":"[7]"}],"fun_headline_variants":["LLMs falter at relational DB construction, new benchmark","Relation-level extraction is LLMs' weak spot, says Doc2DB-Bench","Beyond tables: new benchmark tests document-to-database fidelity","LLMs score 80% on cells but 65-72% on relational links","New benchmark: 203 docs, 42 schemas, tests DB construction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground truth is reliable only if the synthesized documents are complete and exclusive with respect to the gold database: every gold value must be recoverable from the text, and no plausible reading of the text may support an extra fact, a property guaranteed by an LLM validator plus human review rather than by independent audit.","fun_headline_variants_meta":{"raw":{"variants":["LLMs falter at relational DB construction, new benchmark","Relation-level extraction is LLMs' weak spot, says Doc2DB-Bench","Beyond tables: new benchmark tests document-to-database fidelity","LLMs score 80% on cells but 65-72% on relational links","New benchmark: 203 docs, 42 schemas, tests DB construction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000812,"raw_usage":{"total_tokens":3580,"prompt_tokens":983,"completion_tokens":2597,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":2501}},"tokens_in":599,"tokens_out":2597,"duration_ms":20253,"temperature":1.0,"reasoning_tokens":2501,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:35:05.079220+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of Doc2DB-Bench documents, give them to annotators who have never seen the gold database, and ask them to extract every fact they can. If any annotator infers a fact that is not in the gold database, or fails to recover a fact that is in it, that instance violates exclusiveness or completeness, and the benchmark's ground truth for that instance is unsound.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the source relational databases used as ground truth for synthesis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies additional source schemas and instances for the reverse-synthesis pipeline."},{"cited_title":"DTBench: A Synthetic Benchmark for Document-to-Table Extraction","cited_arxiv_id":"2602.13812","evidence_quote":"The prior single-table document-to-table benchmark that Doc2DB-Bench explicitly extends."},{"cited_title":"Sadia, Z","cited_arxiv_id":null,"evidence_quote":"The closest prior text-to-relational database task whose single-view limitations motivate the new benchmark."},{"cited_title":"Zhang, Z","cited_arxiv_id":null,"evidence_quote":"Companion paper that formalizes the Doc2DB problem the benchmark evaluates."}],"review_version":1}