{"id":"ae9fbb28-fb2c-48ba-adee-19f6d20e6c1c","arxiv_id":"2607.28662","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"An ontology-guided, deduplication-aware extraction pipeline for heterogeneous documents reports 70–95% search recall and zero false merges on synthetic intelligence corpora.","lead":"This paper describes a production system that uses a small local language model to pull named people, organizations, and relationships out of PDFs, spreadsheets, Office files, and images, guided by slices of a formal ontology retrieved on the fly. The authors report cutting prompt overhead by 94 percent and lifting search recall from roughly 70 to 95 percent with no false merges, on synthetic intelligence documents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline recall and zero-false-merge claims rest on two synthetic development documents with all thresholds tuned on the same corpus; no independent, held-out validation supports transfer to production document streams.","rationale":"The reader's weakest assumption is that the headline numbers, measured on two synthetic development-stage documents with thresholds tuned on the same corpus, may not transfer to real production document streams. That is exactly the load-bearing concern I identify. The paper is transparent about its limitations: Section 9 says results should be read as directional engineering evidence, not statistically validated effect sizes, and Section 12 says the evaluation documents are synthetic and cannot be redistributed. This transparency is a strength, but it does not remove the gap between the abstract's unconditional framing ('Evaluation on intelligence corpora improved search recall...') and the underlying evidence. I checked for internal logical errors that would force rejection and did not find one; the retrieval, deduplication, and cleaning mechanisms are coherently described and the per-stage fixes are plausible. The concern is not that the architecture is wrong, but that the central quantitative claims are not yet independently verifiable or validated on held-out data. My recommendation is unchanged: the paper should remain conditional pending artifact release and external evaluation. I agree with the reader's identification of the weakest assumption.","tokens_in":34000,"tokens_out":4487,"duration_ms":45860,"concrete_test":"Freeze all thresholds as reported in Appendix A. Run the full pipeline on a held-out set of at least 50 real (non-synthetic) heterogeneous documents for which independent annotators have constructed ground-truth entity sets, alias sets, and hard-negative pairs (same-name different-person). Compute search recall before and after Phase 3, count confirmed false merges, and log catalog tokens per document against the static-slice baseline to test the 94% overhead reduction. If the recall gain does not reproduce or any false merge occurs, the headline claims do not transfer beyond the development corpora.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that ontology-guided retrieval plus layered deduplication raises search recall from roughly 70% to 95% with no false merges. The evidence for this claim is confined to Section 8: one 45-page intelligence report (IR-001.pdf) and one 10-page naval summary (JFS), both of which Section 12 identifies as synthetic intelligence-style documents created for system development. Section 9 explicitly states that the similarity-scorer weights, retrieval thresholds (0.72 floor, 0.80 expansion trigger), and indicator-word lists are heuristics tuned on the development corpus. Table 17 reports the 70%→95% recall improvement without describing how recall was measured, what queries were used, or how ground-truth duplicate pairs were constructed, and no significance testing or confidence intervals appear anywhere. The 94% catalog-overhead reduction is likewise a single-'representative document' comparison in Table 2, not a distributional result. The zero-false-merge guarantee is asserted from a corpus that apparently lacks independent hard-negative labels; Section 7.3.7 documents 79 false-positive merge candidates that had to be removed by a bug fix. Because the code and data are proprietary (Section 12), the externally checkable evidence does not yet establish that these results transfer to real, non-synthetic documents. This is the load-bearing assumption behind the strongest claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a production extraction layer for ontology-aligned knowledge graph construction from heterogeneous documents. The system consumes Kafka metadata, routes PDF/spreadsheet/Office/image content through format-specific handlers, performs two-phase LLM extraction with a locally hosted 4-bit Qwen3.5-9B model, and applies a five-stage refinement pipeline including deterministic cleaning, cross-chunk merging, a relationship second pass, six zero-inference deduplication algorithms, and an embedding-based resolution stage with a hard-conflict guard. The distinguishing component is ontology-guided extraction: a Neo4j ontology slice is retrieved live by embedding similarity and injected into the extraction prompt, reducing catalog overhead by roughly 94% relative to static domain slices. The paper reports that this design improves search recall from about 70% to 95% with no false merges, documents seven classes of upstream quality defects and their fixes, and includes an OCR stress test on a synthetic naval document plus a small public benchmark campaign comparing the local model to a cloud model.","tokens_in":34374,"tokens_out":3328,"duration_ms":34408,"significance":"If the reported gains hold beyond the development corpus, the paper makes a useful engineering contribution: live ontology retrieval for extraction-time grounding, a layered rule-based plus embedding deduplication pipeline with a hard-conflict guard, per-page OCR classification, and a detailed threshold reference (Appendix A) that goes beyond most systems papers. The honest documentation of seven real pipeline bugs and their fixes is a strength, as is the explicit listing of hand-tuned thresholds. However, the headline quantitative claims are not backed by independent validation: the evaluations use single synthetic development documents, thresholds are tuned on those same documents, no significance testing or confidence intervals appear, and the code/data are proprietary. The paper itself (Section 9, Section 12) acknowledges these limitations, but the abstract and conclusion present the numbers without those caveats, which is a serious mismatch between evidence and claim.","major_comments":[{"comment":"The headline claim of search recall improving from roughly 70% to 95% rests on a single 45-page synthetic document (IR-001.pdf). Section 9 concedes that all thresholds (0.72 retrieval floor, 0.80 expansion trigger, similarity-scorer weights, indicator lists) were tuned on the development corpus, and Section 12 states the corpora are synthetic and cannot be redistributed. Table 17 does not describe the query set, the ground-truth duplicate pairs, or the exact protocol used to measure recall. As it stands, the reported improvement is a retrospective fit to the development document, not a predictive result. The authors should either provide a held-out evaluation with multiple documents and significance testing, or sharply qualify the claim in the abstract and Section 1 to 'on a development document.'","section":"§8.1, Table 17; §9; §12"},{"comment":"The claim of 'zero false merges' is not supported by the evidence. Section 7.3.7 documents 79 false-positive merge candidates produced by the abbreviation check, fixed only after evaluation; Table 16 shows 79 false-positive dedup candidates before the fix, while Table 17 reports false positives as 0/0. The corpus, being synthetic and lacking independent hard-negative labels, cannot substantiate a zero-false-merge guarantee. The claim should be reworded to 'no false merges observed in the two synthetic test documents after applying the fixes,' not presented as a general property of the system.","section":"§6.2.4, §7.3.7, §8.1 Table 17"},{"comment":"The 94% catalog-overhead reduction is measured on a single 'representative document' (Table 2), with no distribution, sample size, or variance. Similarly, the retrieval refinements in Tables 4 and 5 are illustrated on one document/passage each. Without a corpus-level evaluation, the 94% figure is anecdotal. Please provide either a multi-document measurement with error bars or change the abstract and Section 1 to 'about 94% on a representative document.'","section":"§4, Table 2; §4.1 Tables 4–5"},{"comment":"The OCR ablation (10-page JFS document, chunk sizes 5 and 2) is a single run on one synthetic document, and the benchmark campaign (Section 8.3) uses samples of n=8–50 with the authors' own caveat that results are 'directional rather than statistically significant.' Despite this, the conclusion states that the pipeline 'tripled relationship coverage' and 'cut hallucinated entities from 174 to zero.' These are not robust claims in their current form. The authors should present the OCR and benchmark results as illustrative case studies and remove the generalizing language from the conclusion, or carry out the multi-pass and multi-document evaluation outlined in Table 20.","section":"§8.2, §8.3, §9"}],"minor_comments":[{"comment":"The row 'Graph false merges prevented: 0 → 4–8 per document' is ambiguous: before deduplication, zero false merges were 'prevented' because no deduplication existed; after deduplication, the claim is that 4–8 were prevented. Clarify the direction of the metric and reconcile with the 79 false-positive candidates reported in Table 16.","section":"§8.1, Table 17"},{"comment":"The text says 'four domain-informed checks' but the list has four items; this is fine, but check 4's threshold of 'more than 30%' and 'fewer than 6 distinct types' should appear in Appendix A's threshold table for completeness.","section":"§3.2"},{"comment":"The phrase 'five systematic precision gaps' appears in the text, but the subsequent list and table contain G1–G5; this is internally consistent, but the introductory sentence says 'five' while Table 3 has five rows — good. However, Section 4.1 introduces 'five' and then Section 4 mentions 'four retrieval refinements' in the conclusion; please make the count consistent ('four' in the conclusion should be 'five' if G1–G5 are counted).","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is an engineering report with a heavy reliance on proprietary code and non-redistributable synthetic data. The architecture is interesting and the bug-fix documentation is valuable, but the headline numbers (70→95% recall, zero false merges, 94% overhead reduction) are not independently verifiable and are explicitly tied to the development corpus. In its current form, the paper overstates the generality of its results. If the journal is willing to accept systems papers with 'directional engineering evidence' clearly labeled as such, the required revision is substantial but feasible: align all claims with the actual evaluation scope, move the limitations into the abstract, and provide a reproducibility plan. If the journal requires externally checkable evidence for quantitative claims, rejection or a very large revision would be necessary."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely informative systems paper, but the headline numbers (70→95% recall, zero false merges, 94% overhead drop) are not yet backed by evidence that transfers beyond the two synthetic documents they were tuned on. The paper itself admits this in Section 9; the rest of the report is stronger than the abstract implies.\n\nWhat's new: the live graph retrieval design—term vectors alongside content vectors, subclass expansion on high-confidence parents, predicate full-text search, density-ranked windows, subclass-aware predicate matching—is a coherent and well-motivated combination. The deduplication stack (six zero-inference rules plus an embedding stage with a hard-conflict guard) is thoughtfully engineered, and the paper does real credit by documenting its own bugs: the source-text truncation, the title-prefix duplication, the 79 false-positive abbreviation matches. That kind of transparency is rare and useful.\n\nWhere it's soft: the empirical support for the central claims is thin. The main recall and false-merge numbers come from one 45-page synthetic intelligence report and one 10-page synthetic naval summary. Every threshold and weight was tuned on the development corpus (0.72 floor, 0.80 expansion, similarity weights, indicator lists). There is no held-out validation, no significance testing, no independent ground-truth labels for the dedup pairs, and no code or data release. The 'zero false merges' claim is also in tension with the 79 false-positive candidates the paper documents elsewhere. The public-benchmark phase uses n=8–50 samples and is explicitly directional. So the architecture is plausible, but the quantified benefits are not yet established.\n\nWho should read it: practitioners building similar extraction layers will get value from the design details and the threshold reference. A researcher looking for validated evidence on ontology-guided extraction should treat the results as anecdotal.\n\nRecommendation: this deserves a serious referee—the engineering is detailed and the limitations are stated honestly—but the paper needs external validation before its headline claims can be trusted. Ask for a held-out corpus, significance testing, and ideally an open reference implementation.","headline":"Solid engineering report, but the headline numbers are development-corpus fits, not predictions.","tokens_in":34807,"tokens_out":2280,"would_cite":false,"duration_ms":22191,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that live retrieval of a small ontology slice, injected into an LLM extraction prompt, keeps a 9B local model aligned to a formal schema while cutting prompt overhead by about 94 percent, and that layered zero-inference ded","keywords":["ontology-guided extraction","knowledge graph construction","entity resolution","deduplication","retrieval-augmented generation","per-page OCR classification","LLM extraction","hierarchical ontology retrieval"],"falsifier":"Take a held-out set of real documents from the same domain with hand-built ground truth, run the pipeline with all thresholds frozen, and measure recall and count false merges; if recall falls well below 95 percent or any false merge appears, the zero-false-merge guarantee and the transferability premise fail.","tokens_in":1382,"feed_emoji":"🕸️","tokens_out":1485,"duration_ms":38137,"temperature":0.7,"pith_summary":"The authors aim to show that a production extraction layer can turn a live, heterogeneous document stream into a validated, ontology-aligned knowledge graph using a modest locally hosted LLM, provided the model is steered by live graph retrieval and repaired by layered deduplication. Unguided LLM extraction fractures type vocabularies and duplicates entities; the system constrains and repairs the model at every stage. If correct, ontology-guided retrieval plus six rule-based deduplication algorithms plus an embedding-based resolution stage with an unoverridable conflict guard is sufficient to make a 9B model production-viable for schema-governed graph construction. The headline numbers are measured on two synthetic development documents, with all thresholds tuned on those documents; the paper frames them as directional engineering evidence, not statistically validated effect sizes.","feed_headline":"Ontology guidance lifts search recall to ~95 percent","feed_subtitle":"Live graph retrieval slashes prompt overhead 94%, and layered dedup keeps false merges at zero.","key_machinery":"The load-bearing mechanism is ontology-guided retrieval: a content sample is split into overlapping windows, embedded, and used to query a vector index over ontology class and predicate definitions; candidates are scored by cosine similarity penalized by PageRank centrality, thresholded at 0.72, and trimmed to a token budget, then injected into the extraction prompt. Five refinements close precision gaps: term vectors alongside prose vectors, one-hop subclass expansion for high-confidence classes, full-text predicate search, density-ranked paragraph windows, and subclass-aware predicate matching. On the deduplication side, the pivotal design is the hard-conflict guard: two entities carrying","core_discovery":"Ontology grounding at extraction time, implemented as live vector retrieval of relevant classes and predicates from a graph database and injection of only those into the prompt, replaces static domain catalog slices and cuts catalog prompt overhead by roughly 94 percent. Combined with a five-stage refinement pipeline (deterministic cleaning, cross-chunk merging, quality-gated relationship second pass, six rule-based deduplication algorithms, and an embedding-based resolution engine whose hard-conflict guard no similarity score can override), search recall rises from roughly 70 to 95 percent while maintaining zero false merges. The paper also demonstrates that per-page OCR classification rout","pith_inferences":["The retrieval recipe (term vectors, subclass expansion, density-ranked windows, and subclass-aware predicate matching) is a general pattern for any embedding-based retrieval over a hierarchical schema, not only for intelligence documents.","The paper implicitly suggests that conservative synonym snapping (a 0.80 cosine floor) is the right trade-off to avoid semantic magnets; this is a testable hypothesis for other ontology-grounded extraction tasks.","The 58.8 percent OCR recall ceiling on the naval document suggests that the binding constraint for long-tail vocabulary is the vision model's reading ability, not the ontology layer; better OCR or multi-pass union would likely be the next gain.","If the architecture generalizes, it offers a template for compliance monitoring and investigative journalism: any governed-schema document stream becomes a queryable graph with auditable merge decisions."],"forward_implications":["A small, locally hosted 9B model can achieve ontology-aligned extraction and high search recall when prompted with a dynamically retrieved ontology slice, without relying on cloud-scale models.","Reclaiming roughly 10,500 tokens per extraction call from a ~11,200-token static catalog allows more source text to fit in a 32K-token window, reducing chunk fragmentation and cross-chunk merge errors.","The hard-conflict guard, combined with context-validated deduplication, makes the reported zero-false-merge result a property of the architecture rather than a lucky threshold choice.","Per-page OCR classification eliminates the information loss of binary text-or-OCR routing for mixed documents, a common real-world case.","Each pipeline stage addresses a distinct quality-defect class, so components can be adopted independently by other extraction systems."],"fun_headline_variants":["Ontology grounding lifts recall to 95% with zero false merges","Live ontology retrieval cuts prompt overhead 94%","Five-stage refinement: recall 95%, zero false merges","Ontology-guided pipeline: 94% less overhead, 95% recall"],"cache_read_input_tokens":36096,"weakest_assumption_plain":"The headline recall and zero-false-merge numbers are measured on two synthetic development documents with all thresholds tuned on them, so the load-bearing assumption is that behavior transfers to real production document streams.","fun_headline_variants_meta":{"raw":{"variants":["Ontology grounding lifts recall to 95% with zero false merges","Live ontology retrieval cuts prompt overhead 94%","Five-stage refinement: recall 95%, zero false merges","Ontology-guided pipeline: 94% less overhead, 95% recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000548,"raw_usage":{"total_tokens":2468,"prompt_tokens":770,"completion_tokens":1698,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":1625}},"tokens_in":514,"tokens_out":1698,"duration_ms":11909,"temperature":1.0,"reasoning_tokens":1625,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T00:44:08.597627+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of real documents from the same domain with hand-built ground truth, run the pipeline with all thresholds frozen, and measure recall and count false merges; if recall falls well below 95 percent or any false merge appears, the zero-false-merge guarantee and the transferability premise fail.","supporting_citations":[],"review_version":1}