{"id":"2bef6176-20c6-4b1f-9e0d-580fd5ade2d5","arxiv_id":"2509.10448","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MatSKRAFT, a graph-neural-network pipeline, extracts material properties and compositions from scientific tables with higher F1 than LLMs and builds a 535k-entry knowledge base from 47k papers.","lead":"MatSKRAFT automatically reads materials-property and composition tables from scientific papers using graph neural networks, and its creators used it to build a database of more than 535,000 entries from nearly 47,000 publications. It reports higher extraction accuracy than large language models while running far faster on a single GPU, but the claimed novelty of 104,000 new compositions is not yet independently validated.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 535k-entry KB and 104k-new-compositions claims rest on unverified extrapolation from 368/737 annotated tables to 68,933 deployed tables; novelty is defined only by absence from two glass databases, so the central database claim lacks a corpus-level audit.","rationale":"The reader's weakest-assumption analysis identifies the representativeness of the manually annotated dev/test sets for the 68,933-table corpus, and the current stress-test concurs: this is the most load-bearing point in the paper. The paper's strongest claim is not just that a GNN beats LLMs on a benchmark, but that running the pipeline on nearly 69,000 tables yields a comprehensive, broadly useful knowledge base with 535,000+ entries and 104,000 new compositions. Every downstream application in Section 2.5—rare-material identification, multi-property screening, temporal trend analysis—inherits the accuracy of the unvalidated KB. If the deployed corpus contains table layouts, journal styles, or linking failures not represented in the small test sets, the reported F1 scores do not guarantee KB precision, and the '104,000 new compositions' claim is especially fragile because it is defined only by absence from two glass-specific databases. The concern is not an internal inconsistency in the GNN architecture or the LLM comparison; those parts are credible, with ablations, detailed prompts, and released code. The soft spot is the external-validity step from test-set metrics to corpus-level claims. The proposed audit is concrete and would settle it: a stratified manual check of KB entries and an independent check of the 'new composition' claim against broader databases. If the audit confirms the reported accuracy and novelty, the central claim stands; if not, the database-scale conclusions would need to be substantially weakened. The verdict remains CONDITIONAL because the paper already identifies some limitations and the reader's conditional recommendation is the appropriate response to an otherwise strong empirical study whose headline scale claims are not yet verified.","tokens_in":40716,"tokens_out":4520,"duration_ms":45006,"concrete_test":"Download the released KB and the source XML/PDF corpus from Zenodo/GitHub; draw a stratified random sample of 500 composition-property pair entries, stratified by journal, table type (SCC/MCC/PI), and intra- vs inter-table linking. Independently verify each entry against the original article, checking property label, value, unit, composition, and link correctness. Compute the resulting precision and compare it with the reported 78.08% pair-level precision and the per-entity F1 estimates. Separately, take 100 of the 104,000 claimed 'new' compositions and query them against Materials Project, ICSD, and a targeted literature search in addition to INTERGLAD/SciGlass; if fewer than 70% remain absent from all of these, the novelty claim fails. A pre-registered audit script with a commit hash would settle whether the KB-level claims survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.3 reports F1=88.68 for property extraction and 71.35 for composition extraction on small manually annotated test sets (368 property tables, 737 composition tables). Section 2.4 then scales to 68,933 tables from 47,242 articles and claims 535,643 KB entries, 100,481 linked composition-property pairs, and more than 104,000 compositions absent from existing databases. For the central claim to hold, the test-set accuracies must transfer to the full corpus. The paper provides no evidence for this: there is no description of how test tables were sampled from the deployed journal corpus, no stratified manual audit of the final KB, and no confidence intervals around any point estimate. The only KB-level accuracy number given, 78.08% pair-level precision, is a single unvalidated estimate. Moreover, the abstract itself says the database is 'pending manual validation,' which is an explicit acknowledgement that the headline database claims are not yet verified. Additional components that act only at scale—XML parsing failures, cross-table identifier linking, duplicate handling, unit normalization—are not covered by the per-entity F1 scores reported in Section 2.3. The '104,000 new compositions' claim is also weaker than it appears: novelty is defined only as absence from INTERGLAD and SciGlass, both glass-specific databases, while the corpus includes Ceramics International, Thin Solid Films, and other non-glass venues. Absence from glass databases is therefore expected for many entries and cannot by itself support a claim of expanding beyond major materials databases. Finally, the abstract/body discrepancies in the headline numbers (66,267 vs 68,933 tables; 509,281 vs 535,643 entries; F1 89.33 vs 88.68) reinforce that these numbers are not stable enough to carry the discovery claims made in Sections 2.5 and 3.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Matskraft, a pipeline for extracting materials compositions and properties from scientific tables. The method converts tables into graphs processed by constraint-driven graph attention networks, with automated training-data generation via distant supervision, property-specific annotation algorithms, and power-law-guided augmentation. The authors report F1 scores of 88.68 for property extraction and 71.35 for composition extraction on manually annotated test sets, outperform five LLM baselines, and claim a 19–496x speed advantage. They then apply the pipeline to roughly 69,000 tables from more than 47,000 articles, constructing a knowledge base of about 535,000 entries, including more than 104,000 compositions reported as absent from INTERGLAD and SciGlass, and demonstrate several materials-discovery applications.","tokens_in":40896,"tokens_out":4471,"duration_ms":41892,"significance":"If the claims hold, this is a substantial contribution: it provides evidence that a specialized, modest-compute GNN pipeline can outperform general-purpose LLMs on structured materials-table extraction, and it demonstrates a path toward automated large-scale knowledge-base construction from the literature. The work is strengthened by extensive ablations, expert-annotated evaluation sets, algorithmic appendices that describe the annotation and post-processing logic, and public code/data links. The main significance risk is that the headline claims about the constructed database and its novelty rest on extrapolation from small test sets to a large, unvalidated corpus.","major_comments":[{"comment":"The headline numbers are internally inconsistent. The abstract states property-extraction F1 of 89.33 and a speed advantage of 6–496x, while the body (Section 2.3.1, Section 2.3.6, and Table C.1) reports 88.68 and 19–496x. The abstract also reports 66,267 tables from more than 45,500 publications yielding 509,281 entries, whereas Section 2.4 and Table E.1 report 68,933 tables from 47,242 articles yielding 535,643 entries. The authors should reconcile the abstract with the body, since these are precisely the results a reader will quote.","section":"Abstract and Section 2.3"},{"comment":"The database-scale claims are not supported by the evaluation presented. The reported F1, precision, and recall figures come from 368 annotated property tables and 737 annotated composition tables (Appendix A.1), but no evidence is given that these tables are representative of the 68,933 tables in the deployed corpus, and no corpus-level audit, stratified sample, or confidence interval is provided. The abstract itself concedes that the database is \"pending manual validation.\" Consequently, statements that the pipeline constructs a \"comprehensive database\" and that the reported accuracies transfer to the full corpus are extrapolations, not demonstrated results.","section":"Section 2.4 and Appendix A.1"},{"comment":"The \"104,000 compositions absent from existing databases\" claim is much weaker than stated. Novelty is defined only as absence from INTERGLAD and SciGlass, both glass-specific databases, whereas the corpus deliberately includes journals such as Ceramics International, Thin Solid Films, Materials Letters, and others covering non-glass systems (Appendix E.1). Finding compositions absent from two glass databases is therefore expected for a large fraction of the corpus. The claim would need comparison against broader materials databases (e.g., ICSD, Materials Project, or a similarly general repository) or, at minimum, a re-framing as \"absent from the two glass databases used here.\" No detail is given about composition canonicalization, duplicate handling, or matching thresholds for this comparison.","section":"Section 2.4 and Appendix E.3"},{"comment":"The composition-extraction recall of 62.97% overall, and 38.93% on Partial-Information tables, is a load-bearing limitation for the large-scale database claim. PI tables constitute a non-negligible fraction of the test and dev splits (111/737 test tables, 109/738 dev tables in Table A.2), and the deployed corpus is processed by the same models. With recall below 40% on this table type, the resulting database will substantially undercount compositions that require contextual inference. The paper should quantify how PI-table recall affects completeness of the 535k-entry knowledge base and should temper the \"comprehensive\" wording accordingly.","section":"Section 2.3.2 and Table A.2"},{"comment":"The provenance of the pair-level precision figure of 78.08% is unclear. Section 2.3.4 describes an evaluation of linked composition-property entities but does not specify the sample size, annotation protocol, or whether the evaluation is on the test set; Appendix C then states that this precision applies \"across our comprehensive knowledge base of over 535,000 entries.\" If 78.08% is a test-set number, it should not be presented as a database-level estimate without a corpus audit. If it is intended as a database-level number, the current text provides no evidence for it.","section":"Section 2.3.4 and Appendix C"}],"minor_comments":[{"comment":"The citation to MatSciBERT appears as \"MatSciBERT [?]\" in Equation (1); the reference marker should be resolved.","section":"Equation (1)"},{"comment":"The framework name is inconsistently capitalized as \"MatSKRAFT\" in some places and \"Matskraft\" in others; the acronym is also expanded only in the Introduction, not at first use in the Abstract.","section":"Throughout"},{"comment":"The Poisson ratio unit accuracy is reported as \"—\" without explanation; a footnote stating that this property is dimensionless would avoid confusion.","section":"Table C.2"},{"comment":"The text refers to removing the \"Annotation Algorithm\" in one ablation label while the corresponding row in Table B.1 is labeled \"w/o Annotation algorithms\"; the nomenclature should be made consistent.","section":"Appendix B.1.2"},{"comment":"The temporal analysis is derived from MatSciBERT analysis of abstracts but no evaluation or validation of that analysis is provided; a brief statement about the source or precision of this application would improve clarity.","section":"Section 2.5 and Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's core extraction models and ablations are solid and the test-set evaluations are carefully designed, but the large-scale database claims are the main risk. The editor should ask the authors to either add a corpus-level validation study, including a stratified manual audit and uncertainty quantification, or substantially soften the database-scale claims. The abstract/body numerical inconsistencies also need correction before the paper can be considered publication-ready."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful systems paper with rigorous test-set evaluation, but the large-scale database claims are ahead of the evidence.\n\nThe real contribution is the pipeline: distant supervision from INTERGLAD, property-specific annotation algorithms, power-law augmentation, and constraint-driven GNNs, all assembled and ablated carefully. The LLM comparison is well done — they gave the baselines generous prompts, and the 15-point F1 gap on property extraction is credible. The 19–496x speedup is a practical advantage that matters for scaling. They also released code and data, which is more than most papers in this space do.\n\nThe soft spot is the database. The 104k \"new compositions\" are only checked against INTERGLAD and SciGlass, both glass-specific, while the corpus includes ceramics and thin-film journals — so absence from those DBs is largely expected. The abstract itself says \"pending manual validation,\" and there is no corpus-level audit: the 368/737 test tables are not shown to be representative of the 68,933 processed tables. That is a load-bearing gap for any discovery claim.\n\nAlso annoying: the arXiv abstract disagrees with the body on F1 (89.33 vs 88.68), speedup (6–496x vs 19–496x), and table/entry counts. That suggests a version mismatch that will confuse readers. And they never report DiSCoMaT's own numbers on the new test split, despite building on it, so the improvement over the closest baseline is unquantified.\n\nThe quality of the work is otherwise good: ablations are extensive, the unit-extraction and post-processing details are transparent, and the Discussion honestly notes the composition-recall weaknesses. Given that, I would accept this for peer review if the editor pushes for major revision.\n\nWho this is for: people building materials-knowledge bases or table-extraction systems. The method itself is reusable; the database claims need verification before using the KB.","headline":"A solid, carefully evaluated table-extraction system with genuine engineering value, but the database-scale claims run ahead of the evidence.","tokens_in":41657,"tokens_out":3875,"would_cite":true,"duration_ms":34571,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A specialized graph-neural-network pipeline extracts materials data from scientific tables with higher F1 scores than leading large language models, while running far faster on a single GPU.","keywords":["materials informatics","scientific table extraction","graph neural networks","property extraction","composition extraction","large language models","knowledge base construction","distant supervision"],"falsifier":"Manually annotate a random sample of, say, 300 tables from the 68,933-table corpus and run the released pipeline on them; if observed F1 falls substantially below the reported 88.68 for properties or 71.35 for compositions on the held-out test sets, the extrapolation fails. A second check is to take 100 of the claimed 104,000 “new” compositions and verify by reading the source paper that the composition truly appears and belongs to the linked property record.","tokens_in":40382,"feed_emoji":"🧪","tokens_out":5255,"duration_ms":44851,"temperature":0.7,"pith_summary":"MatSKRAFT is a five-stage pipeline that converts scientific tables into graphs and uses specialized graph neural networks with rule-based post-processing to extract material compositions and physical properties. The paper reports F1 scores of 88.68 for property extraction and 71.35 for composition extraction on manually annotated test sets, beating several frontier LLMs by 15–17 F1 points while processing each table in 0.22–0.39 seconds. Applied to nearly 69,000 tables from more than 47,000 papers, the framework reportedly produced a knowledge base of over 535,000 entries, including more than 104,000 compositions absent from established databases. If these claims hold, the practical significance is that corpus-scale literature mining could become a routine, inexpensive task for a single research group rather than a costly LLM-based operation.","feed_headline":"A 1-GPU system extracts materials tables at higher F1 than LLMs","feed_subtitle":"MatSKRAFT turns 69,000 tables into 535k structured entries, 19–496x faster than language-model baselines.","key_machinery":"The load-bearing object is the table-as-graph representation fed to a two-layer graph attention network (GAT). Nodes are cells, row/column headers, and the caption; edges link cells to their row and column headers and feed caption information into headers. Four differentiable constraints—material-property association, material-identifier exclusivity, property exclusivity, and identifier uniqueness—are added to the cross-entropy loss with weight λ = 50 so the network learns scientifically coherent table readings. A separate modular post-processing layer then performs unit extraction (including over 15 electrical-conductivity unit variants), physical-range checks, exponent reconstruction for values like thermal expansion, and disambiguation of overloaded symbols. This combination, rather than any single component, carries the argument: removing post-processing costs 9.38 F1 points, and removing the annotation algorithms costs 9.02 F1 points.","core_discovery":"The paper's central claim is that a deliberately specialized, constraint-driven extraction architecture can outperform generalist large language models on scientific table understanding at a fraction of the compute cost. The framework represents each table as a graph with nodes for cells, headers, and captions, and trains graph attention networks under soft constraints that encode conventions of materials tables—for example, that a material identifier and its properties should occupy the same orientation, and that only one header should identify the material. Domain rules then post-process the network's predictions, validating units, physical ranges, and ambiguous notation such as “n” standing for refractive index or Poisson's ratio. On expert-annotated test sets the authors report F1 88.68 across 18 properties, with density at 96.50 and glass transition temperature at 93.00, and F1 71.35 for compositions, with single-cell and multi-cell tables at 78.62 and 75.99. They also run the pipeline over the large corpus, building 100,481 linked composition–property pairs and a knowledge base of 535,643 entries, with the large-scale database still pending manual validation.","pith_inferences":["The largest untested step is the leap from 368 and 737 manually annotated test tables to roughly 69,000 deployed tables; a random manual audit of even 200 deployed tables would tell whether the measured F1 holds outside the dev/test distribution.","Because the framework is fully automated and works without commercial database supervision (the no-distant-supervision ablation reaches 93.04 precision), a natural extension is retraining it for battery, polymer, or catalysis tables, where property vocabularies differ.","The 104,000 “new” compositions are defined as absent from INTERGLAD and SciGlass; some may appear in other aggregated databases or in text-only form in the same papers, so checking against further databases would clarify how much genuinely new composition space is covered.","Because evaluation requires exact value+unit+name matches, the reported entity-level F1 is conservative; however, linked composition–property pairs reach only 78.08 precision, so downstream users should treat pair-level claims with more caution than single-entity extraction."],"forward_implications":["If the reported F1 scores transfer to the full corpus, researchers can extract structured composition–property data from entire journal collections on a single GPU in hours rather than weeks.","The reported 104,000 compositions absent from existing databases would substantially widen the known inorganic composition space, especially for rare-earth elements, transition metals, and semiconductor dopants.","The pipeline's 0.22–0.39 seconds per table makes iterative, corpus-wide information extraction economical enough for periodic re-runs as new literature is published.","The ablation results suggest that the same design—constraint-driven GNN plus domain post-processing—could be ported to other fields whose tables follow systematic reporting conventions."],"supporting_citations":[{"why":"Supplies the base architecture and table-type taxonomy (SCC, MCC, PI) for composition extraction, which MatSKRAFT enhances.","marker":"[13]"},{"why":"INTERGLAD provides the distant-supervision source that generates initial training labels and is also a comparison database for coverage claims.","marker":"[39]"},{"why":"SciGlass serves as the other reference database against which the 104,000 newly covered compositions are defined.","marker":"[45]"},{"why":"MatSciBERT supplies the embeddings used to initialize cell and caption nodes in the property-extraction graph.","marker":"[10]"},{"why":"The MIT table parser converts raw XML tables into the two-dimensional lists that feed the extraction pipeline.","marker":"[8]"},{"why":"Documents the heterogeneity and ambiguity challenges in materials information extraction that motivate the constraint-driven and post-processing design.","marker":"[7]"}],"fun_headline_variants":["MatSKRAFT beats LLMs on materials tables, 19–496x faster","Graph-based extraction outshines LLMs on materials data","One GPU, higher F1 than LLMs: MatSKRAFT for materials tables","66k tables, 509k entries: MatSKRAFT outperforms LLMs","Materials knowledge extraction: MatSKRAFT tops LLMs, 6–496x speedup"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole-scale database and discovery claims rest on the assumption that the manually annotated dev and test tables are representative of the roughly 69,000 tables in the large-scale run, so the measured F1, precision, and recall transfer to the full corpus.","fun_headline_variants_meta":{"raw":{"variants":["MatSKRAFT beats LLMs on materials tables, 19–496x faster","Graph-based extraction outshines LLMs on materials data","One GPU, higher F1 than LLMs: MatSKRAFT for materials tables","66k tables, 509k entries: MatSKRAFT outperforms LLMs","Materials knowledge extraction: MatSKRAFT tops LLMs, 6–496x speedup"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1567,"prompt_tokens":968,"completion_tokens":599,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":496}},"tokens_in":584,"tokens_out":599,"duration_ms":5449,"temperature":1.0,"reasoning_tokens":496,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:54:14.540225+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually annotate a random sample of, say, 300 tables from the 68,933-table corpus and run the released pipeline on them; if observed F1 falls substantially below the reported 88.68 for properties or 71.35 for compositions on the held-out test sets, the extrapolation fails. A second check is to take 100 of the claimed 104,000 “new” compositions and verify by reading the source paper that the composition truly appears and belongs to the linked property record.","supporting_citations":[{"cited_title":"International glass database system, March 2019","cited_arxiv_id":null,"evidence_quote":"INTERGLAD provides the distant-supervision source that generates initial training labels and is also a comparison database for coverage claims."},{"cited_title":"Publisher: EPAM Systems","cited_arxiv_id":null,"evidence_quote":"SciGlass serves as the other reference database against which the 104,000 newly covered compositions are defined."}],"review_version":2}