{"id":"f45f56a4-3e98-468d-acc5-d60e234a7f04","arxiv_id":"2508.20543","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"SemDR applies a Group Steiner Tree heuristic to a domain-enriched concept graph for agriculture document retrieval and reports large gains over Lucene, ElasticSearch, and Doc2Vec.","lead":"An agriculture-domain document retrieval system, SemDR, uses a Group Steiner Tree algorithm over a domain knowledge graph to identify relevant concepts and fetch documents. The authors report 90% precision and 82% accuracy on 170 expert queries, but the evaluation has several methodological problems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 90% precision / 82% accuracy is not an independent measure: the reference solution, the domain knowledge, and the 0.9 threshold all come from the same expert community, so the evaluation cannot separate retrieval quality from alignment with the system's own construction.","rationale":"I read the paper as making a systems claim: SemDR, using a group-Steiner-tree search over an expert-built agriculture concept graph, retrieves documents with roughly 90% precision and 82% accuracy, far above Lucene, ElasticSearch, and Doc2Vec on 170 government-agriculture queries. For this claim to hold, the evaluation must measure retrieval quality against judgments that are not themselves shaped by the same expert choices that define SemDR. That condition fails. Section V-A's reference solution is produced by domain experts who also supply the domain knowledge (Section IV-A), define 104 of the 170 queries (Table III QS5), and influence the 0.9 semantic-proximity threshold (Section III, footnote 2). The paper even says the reference solution is 'built considering the application of the proposed research.' Thus the reported scores are better read as a consistency check between the experts' mental model and a system built from that model, not as an independent benchmark. The swapped precision/accuracy figures between Abstract and Conclusion, the absence of error bars or per-query breakdowns, and the lack of released data and code make it impossible to re-derive or verify the headline numbers. I do not see an internal mathematical flaw in the GST formulation or complexity analysis itself; the weak point is entirely in the evidence supporting the empirical claim. Because the reader's REJECT is based on the same lack of independent, reproducible evaluation, I would keep the verdict unchanged.","tokens_in":17798,"tokens_out":4590,"duration_ms":44185,"concrete_test":"Hold out a random 50-query subset of the 170 queries before any tuning. Have independent domain experts who did not contribute to the concept graph or the threshold produce relevance judgments for those queries on the full corpus, using pooling across all systems. Fix the 0.9 threshold and all graph weights using only the remaining 120 queries, then run SemDR and the three baselines on the held-out queries with the independent judgments. If SemDR's precision/accuracy on the held-out set falls below the reported 90/82, or if it does not exceed Elasticsearch by a statistically significant margin on the same independent judgments, then the reported advantage is an artifact of reference alignment rather than demonstrated retrieval quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the empirical result in Section V-D: SemDR precision ~90%, recall ~88%, accuracy 82%, F1 89%. The load-bearing assumption is that the reference solution measures independent retrieval quality. That assumption is not met. Section V-A says the reference solution is 'built considering the application of the proposed research' and that domain experts selected files for all search strings. Those same experts (or at least the same expert community, with no demonstrated separation) supply the domain knowledge from which the semantic concept graph is constructed (Section IV-A), provide 104 of the 170 queries (QS5 in Table III), and their 'insights' were used with experimentation to set the semantic proximity threshold of 0.9 (Section III, footnote 2). The GST heuristic then finds paths among anchors that are defined by that expert-derived graph and threshold. So SemDR is effectively tuned to reproduce the experts' file selections, while Lucene, ElasticSearch, and Doc2Vec are generic baselines with no such alignment. A high score under this protocol could reflect expert-consistent concept engineering rather than superior retrieval. The swapped precision/accuracy numbers (90% precision in the Abstract vs. 90% accuracy in the Conclusion) and the lack of code/data compound the problem, but the circular reference is the decisive issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SemDR, a semantic document retrieval system that builds a domain-specific concept graph enriched with latent concepts, identifies anchor concepts for a query using a semantic proximity threshold, and retrieves documents by applying a Group Steiner Tree (GST) heuristic to connect those anchors. The system is implemented over heterogeneous Karnataka agriculture data and evaluated on 170 real-world search queries against Lucene, ElasticSearch, and Doc2Vec. The authors report that SemDR achieves precision 90%, recall 88%, accuracy 82%, and F1-score 89% (Section V-D), claiming substantial gains over the baselines. The paper also contributes a complexity analysis of the proposed heuristic and a discussion of domain and geographic ontology construction. The central claim is empirical: that SemDR's retrieval quality is superior to the baselines on this benchmark.","tokens_in":18084,"tokens_out":3346,"duration_ms":35105,"significance":"If the reported evaluation were independent, the paper would provide a useful application of the Group Steiner Tree formalism to document retrieval, and the effort to combine domain knowledge, latent concept grouping, and heterogeneous data indexing is a worthwhile engineering contribution. The paper also includes an explicit complexity analysis and a concrete real-world deployment scenario. However, the central empirical claim is not currently established: the reference solution, the domain knowledge graph, 104 of the 170 test queries (QS5), and the 0.9 semantic threshold all originate from the same expert community with no demonstrated separation, so the reported precision/recall numbers cannot be read as independent measures of retrieval quality. The numerical headline is also internally inconsistent across the abstract, Section V-D, and the conclusion. No code, data, or statistical detail is provided to allow independent verification. The contribution is therefore assessed as not yet suitable for publication as a validated retrieval result.","major_comments":[{"comment":"The evaluation is circular. The reference solution is built \"considering the application of the proposed research\" and requires domain experts to select files for every search string (Section V-A). Those same experts, or the same expert community, are the source of the domain knowledge from which the semantic concept graph is constructed (Section IV-A), they provide 104 of the 170 queries (QS5 in Table III), and their insights were used in setting the 0.9 semantic proximity threshold (Section III, footnote 2). Under this protocol, SemDR's anchors, latent groups, and query set are all aligned with the experts' own file selections, while Lucene, ElasticSearch, and Doc2Vec are generic baselines with no such alignment. The reported 90% precision is therefore not an independent measure of retrieval quality; it could predominantly reflect expert-consistent concept engineering. A meaningful evaluation would require a held-out relevance judgment set, a separation of the knowledge-graph builders from the relevance judges, and a threshold selected on a development set rather than on the test queries.","section":"Section V-A, Section IV-A, Section III footnote 2, Table III"},{"comment":"The headline numbers are internally inconsistent. The abstract states \"precision and accuracy achieving levels of 90% and 82% respectively\"; Section V-D states \"precision and recall at 90% and 88%, respectively, accuracy at 82%, and an F1-score of 89%\"; and the conclusion states \"attaining an accuracy rate of 90% and a precision score of 82%.\" These are three different statements of the central result. The authors must correct this and report per-query aggregates, not just the global values.","section":"Abstract, Section V-D, Section VII"},{"comment":"The evaluation metrics are not defined in an auditable way. The text defines True Positive (TP) as \"the count of documents retrieved by the manual selection process and other baseline systems,\" which mixes the reference with the systems being evaluated, and it defines True Negative (TN) as documents retrieved by neither, which is undefined for an unpooled retrieval setting where most documents are never examined. Accuracy is then computed as (TP+TN)/(total), which is dominated by unjudged negatives and is not a meaningful retrieval measure. The formulas must be replaced with standard IR definitions based on pooled relevance judgments, and the pooling protocol must be described.","section":"Section V-C"},{"comment":"The algorithmic contribution is underspecified. Algorithm 2 repeatedly calls a subroutine \"STEINER TREE (G, Li)\" but the paper does not state what this subroutine computes (exact Steiner tree? a shortest-path heuristic?) and does not specify how trees are merged when common vertices are found. The complexity analysis in Section III-B assumes an exact Steiner tree step of O(2^k n^2), yet the text describes a greedy merging heuristic; the claimed overall complexity O(k * 2^k * n^2) is therefore not derived from the pseudocode. Without a precise description of the subroutine and the merging rule, the proposed \"Semantic-based Concept Retrieval using Group Steiner Tree\" cannot be implemented or verified from the manuscript.","section":"Section III and Algorithm 2"},{"comment":"The ranked-retrieval comparison is presented without statistical support. Table V reports percentages for Top-3/5/7/10, but there are no error bars, no standard errors across the 170 queries, and no significance test showing that the differences between SemDR and Lucene/ElasticSearch are not due to query-level variance. Given the circularity concern, these comparisons do not yet support the claim that SemDR retrieves more relevant documents than the baselines.","section":"Section V-D.4 and Table V"}],"minor_comments":[{"comment":"The formula for the mean Type-2 error is written as a sum over search queries of (Reference solution - retrieved documents)/reference solution, but no averaging divisor is shown and the connection to the values in Table IV is not stated precisely. Please define the quantity formally.","section":"Section V-D.2"},{"comment":"The first column of Table V is labeled \"SDR\" rather than \"SemDR,\" and the equation above the table contains a malformed absolute-value expression. These should be corrected.","section":"Section V-D.4 and Table V"},{"comment":"Wu-Palmer similarity is defined for WordNet synsets, but the paper uses it on a custom agriculture concept graph without stating how the concepts are mapped to an ontology or how similarity is computed for concepts not present in WordNet. This should be clarified because the anchor selection depends on it.","section":"Section II and Section III"},{"comment":"The text says \"Figure 6 illustrates the True Positive value over different baseline systems,\" but Figure 6 is described as showing precision, recall, accuracy, and F1-score; the figure captions and in-text references need to be aligned.","section":"Section V-D.3"},{"comment":"The paper does not provide the code, the constructed concept graph, or the query/reference files. For a system whose central claim is empirical, sharing these artifacts or an online appendix would be important for reproducibility and for enabling a non-expert to audit the evaluation.","section":"General"}],"recommendation":"reject","confidential_remarks":"The decisive issue is the circular evaluation: the same expert community supplies the reference solution, the domain knowledge graph, the majority of the test queries, and the semantic threshold, so the reported superiority over generic baselines is not an independent finding. The internal inconsistency of the headline precision/accuracy numbers and the lack of code or data strengthen the case for rejection. I would not advise inviting a revision unless the authors can provide a genuinely external relevance judgment set and a threshold chosen without access to the test queries; that is a substantial reworking of the evaluation, not a local fix."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know about this paper because it is a good example of a real system whose reported results are undermined by a circular evaluation. SemDR is a document retrieval system for heterogeneous agriculture data (SQL, CSV, PDFs, images) that builds a domain-specific concept graph, groups concepts into latent semantic groups, and uses a Group Steiner Tree heuristic to find concepts relevant to a query. The application is genuine, and the system was tested on 170 real queries against Lucene, ElasticSearch, and Doc2Vec. If the numbers were true—90% precision, 88% recall—it would be a solid result for a government data portal.\n\nThe core idea is sensible: use domain knowledge to enrich a concept graph, then treat query concepts as terminal groups and find a low-cost tree connecting them. The latent grouping (concepts with mutual semantic proximity > 0.9) and the inverse-Jaccard edge weighting are reasonable choices. The paper is clearly written in parts, and the authors did real engineering work—heterogeneous data, semantic mapping, a query module.\n\nThe soft spot is the evaluation. The reference solution was created by domain experts selecting relevant files for each query. The same expert community (with no demonstrated separation) supplies the domain knowledge from which the concept graph is built, and the semantic threshold of 0.9 was tuned after experimenting and 'taking into account the insights of domain experts' (footnote 2). So the system is effectively calibrated to reproduce the experts' selections, while the baselines are generic and get no such alignment. Under that protocol, high precision may just mean the system matches the experts' mental model—not that it retrieves better. That is the load-bearing flaw.\n\nThere are smaller issues too: the abstract says precision 90% / accuracy 82%, while the conclusion swaps them (accuracy 90% / precision 82%). Section V-C has a typo in the confusion matrix labels (calls TP 'TN'). No code or data is released, no error bars or significance tests are given, and the novelty is incremental—Steiner trees on knowledge graphs appear in prior work they cite (e.g., [50], [52]-[54]). The new bits are the latent grouping and domain-specific weighting.\n\nAll that said, the paper is not a scam or a toy. The system is real, the idea is coherent, and the problem matters. It deserves a serious referee who will push for an independent ground truth, separated from the knowledge-graph construction, and a proper statistical comparison. As submitted, the numbers cannot be trusted. I would send it to review with that expectation, not desk-reject it.","headline":"A real system with a plausible method, but the headline numbers are not credible due to a circular evaluation.","tokens_in":18604,"tokens_out":3823,"would_cite":false,"duration_ms":35338,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68P20","05C85"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that routing document retrieval through a domain-specific concept graph and a Group Steiner Tree heuristic lifts precision to 90% and accuracy to 82% on 170 agriculture queries, well above Lucene, ElasticSearch, and…","keywords":["Document Retrieval","Semantic Concept Graph","Group Steiner Tree","Domain Knowledge","Heterogeneous Data Sources","Semantic Proximity","Agriculture Data","Knowledge Graphs"],"falsifier":"Have a separate group of domain experts, blind to the concept graph and threshold, independently label relevant documents for the same 170 queries, then recompute precision; if SemDR's score falls toward or below the baselines, the expert-alignment explanation stands.","tokens_in":17582,"feed_emoji":"📄","tokens_out":10704,"duration_ms":91085,"temperature":0.7,"pith_summary":"The paper sets out to show that document retrieval gains from domain knowledge if queries are translated into concept nodes on a graph. It proposes SemDR, a system that builds a weighted semantic concept graph from agriculture domain knowledge, marks query words as anchor concepts, and uses a Group Steiner Tree heuristic to find the concept nodes that best connect those anchors. Documents attached to the identified concepts are then retrieved, with ranking by semantic similarity. On 170 real-world agriculture queries, the paper reports precision of 90%, recall of 88%, accuracy of 82%, and an F1-score of 89%, all well above the three baselines tested.","feed_headline":"Semantic graph search hits 90% retrieval precision","feed_subtitle":"Agriculture-domain concept graph plus Group Steiner Tree outperforms Lucene, ElasticSearch and Doc2Vec on 170 queries.","key_machinery":"The Group Steiner Tree on a weighted Semantic Concept Graph G=(C,R,W) is the load-bearing mechanism. The node set C holds direct concepts from domain knowledge plus latent concepts, which are groups of concepts connected by high semantic proximity; edges R are contextual (hierarchical) or semantic relations; and edge weights W are set by an inverse document-overlap score (1 minus Jaccard similarity between the document sets of the two concepts). The algorithm first identifies anchor concepts by semantic proximity to the query, expands each anchor to its latent group, and then greedily computes and merges Steiner trees for the groups, returning the tree's nodes as relevant concepts. This turns the retrieval question—'which concepts tie the query's anchors together?'—into a tractable graph optimization and supplies the document-access path used by SemDR.","core_discovery":"The central claim is that the relevant documents for a query can be found by solving a Group Steiner Tree problem on a domain-specific concept graph. Anchor concepts are graph nodes whose semantic proximity to the search words exceeds a threshold; each anchor is relaxed to a latent group of closely related concepts, and the algorithm repeatedly grows and merges low-cost Steiner trees until every group is touched. The nodes of the final tree are declared relevant concepts, and documents are retrieved through an index that maps documents to concepts. The paper reports that on its benchmark this pipeline reaches 90% precision, 88% recall, 82% accuracy, and 89% F1, with a mean type-2 error of 11.9%, and that its ranked results improve from 75% relevant at top-3 to 89% at top-10 while all baselines decline.","pith_inferences":["Because the expert-built concept graph and the expert-selected reference solution come from the same source, the reported 90% precision likely contains an alignment effect; an independent gold standard with multiple annotators would quantify how much.","The approach's transfer cost is the domain knowledge itself: the GST algorithm is generic, so applying it to a new domain mainly means building a new concept graph and re-tuning the semantic-proximity threshold rather than changing the retrieval logic.","The concept-selection step could act as a semantic router in front of a neural ranking model: retrieve documents attached to the GST-identified concepts, then rerank them with dense or cross-encoder models, potentially combining domain grounding with modern embeddings.","A concrete stress test would use a sparse document collection where inverse-Jaccard weights might favor generic concepts; comparing against term-frequency-aware edge weights would show whether the weighting scheme, not the GST structure, drives the gains."],"forward_implications":["A query whose words never appear literally in any document can still retrieve relevant documents through latent concept groups, covering the indirect-query case.","Heterogeneous data sources can stay in their original structured and unstructured formats because retrieval happens through the semantic index rather than through reformatting the data.","The GST formulation gives a principled reason to prefer concept paths with many shared documents, since the inverse-Jaccard edge weights keep the tree inside concepts with overlapping document sets.","If the reported ranking results hold, users benefit from a semantic ranking that places more relevant documents in the top positions as the considered set grows from top-3 to top-10.","The system's query module permits join-like queries across structured documents, extending the contribution from retrieval to light-weight analysis on the retrieved data."],"supporting_citations":[{"why":"Establishes the concept-graph representation of domain knowledge that SemDR builds on.","marker":"[20]"},{"why":"Provides the semantic proximity measure used to identify anchor concepts and latent concept groups.","marker":"[22]"},{"why":"Supplies the Group Steiner Tree formulation that the retrieval algorithm instantiates.","marker":"[23]"},{"why":"Presents the approximation algorithm whose complexity analysis frames the GST heuristic.","marker":"[24]"},{"why":"Supports the NP-hard complexity claim that motivates a heuristic instead of exact Steiner computation.","marker":"[25]"},{"why":"Confirms the approximation-algorithm background cited alongside [25] for the Steiner hardness argument.","marker":"[26]"},{"why":"Defines the Lucene baseline that SemDR is compared against.","marker":"[27]"},{"why":"Defines the Elasticsearch baseline used in the comparison.","marker":"[29]"},{"why":"Provides the Doc2Vec dense-embedding baseline used in the comparison.","marker":"[30]"}],"fun_headline_variants":["Group Steiner Tree boosts document retrieval to 90% precision","Domain concept graph + Steiner Tree yields 90% precision","Graph-based retrieval hits 90% precision with domain knowledge","Domain-aware Steiner Tree reaches 90% precision for document retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation's ground truth and the system's domain knowledge were both produced by the same domain experts and the 0.9 semantic-proximity threshold was tuned with their input, so the reported 90% precision may reflect alignment with those experts' judgment rather than an independent measure of retrieval quality.","fun_headline_variants_meta":{"raw":{"variants":["Group Steiner Tree boosts document retrieval to 90% precision","Domain concept graph + Steiner Tree yields 90% precision","Graph-based retrieval hits 90% precision with domain knowledge","Domain-aware Steiner Tree reaches 90% precision for document retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000514,"raw_usage":{"total_tokens":2490,"prompt_tokens":930,"completion_tokens":1560,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":1492}},"tokens_in":546,"tokens_out":1560,"duration_ms":8840,"temperature":1.0,"reasoning_tokens":1492,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:43:26.894513+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a separate group of domain experts, blind to the concept graph and threshold, independently label relevant documents for the same 170 queries, then recompute precision; if SemDR's score falls toward or below the baselines, the expert-alignment explanation stands.","supporting_citations":[{"cited_title":"Cognitive retrieve: Empowering document retrieval with semantics and domain-specific knowledge graph,","cited_arxiv_id":null,"evidence_quote":"Establishes the concept-graph representation of domain knowledge that SemDR builds on."},{"cited_title":"Answering Complex Questions by Joining Multi-Document Evidence with Quasi Knowledge Graphs","cited_arxiv_id":"1908.00469","evidence_quote":"Supplies the Group Steiner Tree formulation that the retrieval algorithm instantiates."},{"cited_title":"An approximation algorithm for the group steiner problem,","cited_arxiv_id":null,"evidence_quote":"Presents the approximation algorithm whose complexity analysis frames the GST heuristic."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the NP-hard complexity claim that motivates a heuristic instead of exact Steiner computation."},{"cited_title":"Apache lucene 4,","cited_arxiv_id":null,"evidence_quote":"Defines the Lucene baseline that SemDR is compared against."},{"cited_title":"Gormley and Z","cited_arxiv_id":null,"evidence_quote":"Defines the Elasticsearch baseline used in the comparison."}],"review_version":2}