{"id":"60beb27a-707b-4f9f-9c87-b4b8952fd7bf","arxiv_id":"2509.10087","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A climate knowledge graph built from prior extraction work is presented with example queries, but without evaluation or released artifacts.","lead":"This paper describes a knowledge graph for climate science papers that links models, regions, and weather phenomena, and shows how to query it. It argues this gives more precise literature search than keywords, but provides no test results or a working system.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No query results or extraction-accuracy evidence shown; Cypher listings alone cannot support the claim that the KG enables precise climate retrieval.","rationale":"The reader's weakest assumption is extraction reliability; my concern is broader but includes it: even a perfectly reliable extraction would not salvage the paper without executed query results and quantitative evaluation. The paper positions itself as a use-case paper, not a construction paper, and prior work may contain evaluations, so the absence here could be scope-driven. But the abstract makes unqualified claims of demonstrated capability, and the only demonstrations are unexecuted Cypher fragments and an anecdotal figure comparison. The manuscript also contains placeholder references ([41]–[45], [50]), a duplicated query numbering, and typos, indicating a draft state. In good faith, the core idea is plausible and the authors cite relevant prior work, but the evidence does not support the central claim as stated. Therefore the reader's REJECT verdict stands unchanged; no adjustment is needed.","tokens_in":8884,"tokens_out":3690,"duration_ms":39885,"concrete_test":"Execute Listings 1–3 against the actual ClimatePub4KG (or release a dump) and report result counts plus a random sample of 50 returned paper–entity–location triples per query, manually verified against the source sentences. Also measure precision/recall against a human-annotated gold set for these queries. If any query returns zero rows, or if sampled-triple precision is below roughly 0.8, the central claim that the KG enables precise climate retrieval fails in its demonstrated form.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ClimatePub4KG \"supports structured, semantic queries that help researchers discover precise connections.\" This requires the KG to contain correct entities and relationships, and the queries to return correct results. The manuscript provides neither. Listings 1–3 are Cypher fragments, but no output rows, result counts, or executed query traces are shown; Figure 2 is a schematic, not a query result. The extraction pipelines (ClimateIE, SciER) are cited but no precision/recall is reported here, and the conclusion explicitly states \"Evaluation will be key: we are developing benchmark queries with domain experts to assess performance and usability\"—an admission that no evaluation exists yet. More fundamentally, Query 2 depends on nodes labeled `Model|Project` containing \"CMIP5\" and Location nodes with a Wikidata description containing \"United States\"; Query 3 depends on `TargetsLocation` edges from PNA to US locations. If the underlying relation extraction omitted or mislabeled these edges, the Cypher would silently return empty or incomplete results while looking plausible. The ChatGPT comparison is anecdotal and does not quantify correctness. Thus the abstract's central claim is not supported by evidence in this manuscript.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ClimatePub4KG, a climate-science knowledge graph built from prior extraction systems (ClimateIE, SciER) and a taxonomy-driven construction pipeline. The authors argue that, unlike keyword-based search, this KG supports structured semantic queries that let researchers discover precise connections among climate models, datasets, regions, teleconnection patterns, and papers. The manuscript presents three natural-language queries with Cypher translations (Listings 1–3), a qualitative comparison against ChatGPT-4o for one of these queries, and a discussion of RAG integration, systematic reviews, and future evaluation plans. The central claim is that the KG enables precise, context-aware retrieval for climate research.","tokens_in":9154,"tokens_out":4287,"duration_ms":48640,"significance":"If substantiated, a domain-specific climate KG with reliable entity/relation extraction and semantic query support would be genuinely useful for literature discovery, model evaluation, and RAG grounding. The three query scenarios (cold-air outbreaks over North America, CMIP5+NAO in the Southeast U.S., PNA targeting U.S. locations) are well chosen and illustrate the kind of multi-faceted questions climate researchers ask. However, the paper provides no query results, no precision/recall numbers, no user study, no released graph or code, and no quantitative comparison with baseline retrieval. The only evidence is a single anecdotal ChatGPT comparison. The manuscript explicitly states 'Evaluation will be key' and that benchmark queries are still being developed, which is an admission that the central claim is untested. As a proposal or vision statement the paper has merit, but as a demonstration of a working retrieval backend it is not yet supported.","major_comments":[{"comment":"The central claim that ClimatePub4KG 'supports structured, semantic queries' is not demonstrated. Listings 1–3 show only Cypher fragments; no output rows, result counts, executed query traces, or example result graphs are presented. Figure 2 is referenced as the KG output, but the figure is either absent or schematic and does not provide machine-readable results. The conclusion states, 'Evaluation will be key: we are developing benchmark queries with domain experts to assess performance and usability,' which explicitly defers evaluation. Without any retrieval results, the paper cannot support the abstract's claim of enabling precise discovery.","section":"Use Case: Querying the Climate Knowledge Graph; Conclusion"},{"comment":"The retrieval quality depends entirely on the accuracy of the entity and relation extraction that constructed the graph, but no precision/recall or error analysis is reported here. The paper cites ClimateIE and SciER as the foundation, yet gives no evidence that the required nodes and edges exist with sufficient accuracy in the graph. For example, Listing 2 requires nodes labeled `Model|Project` whose names contain 'CMIP5', a `Teleconnection` node named 'NORTH_ATLANTIC_OSCILLATION', and `Location` nodes with specific Wikidata descriptions; Listing 3 depends on `TargetsLocation` edges from the PNA pattern to U.S. locations. If the extraction mislabeled or omitted these, the Cypher would silently return empty or incomplete results. The manuscript needs at least extraction-quality figures from the cited prior work, or an end-to-end evaluation on a held-out set of queries.","section":"Use Case: Querying the Climate Knowledge Graph; Background"},{"comment":"The ChatGPT comparison is anecdotal and does not provide quantitative or reproducible evidence. It compares a single query (the paper's Query 1) and states that ChatGPT's graph lacks citations and has incorrect relationship directions, but no evaluation protocol, metrics, or full output is shown. The two figures are referenced but not included in the text in a usable form. Even if the ChatGPT output is flawed, one example cannot establish that ClimatePub4KG produces precise, accurate, and complete results. A systematic comparison with a query set and correctness judgments would be required.","section":"A comparison to ChatGPT-4o"},{"comment":"Section 5 claims that the KG offers 'enhanced precision and recall,' supports 'multi-hop queries,' and 'excels' at certain query types, but these are assertions of potential, not demonstrated capabilities. No experiments, baselines, or comparison to keyword search or other KG-based systems are provided. These claims should be reframed as research hypotheses or supported by empirical evidence.","section":"Applications and Implications for IR"}],"minor_comments":[{"comment":"Typography and terminology: 'Cipher' should be 'Cypher'; 'Devloper' should be 'Developer'; 'Enviromentalist' should be 'Environmentalist'; 'combining through' should be 'combing through'; 'Here were present example queries' should be 'Here we present example queries.'","section":"Use Case: Querying the Climate Knowledge Graph"},{"comment":"The query numbering is inconsistent: 'Natural Language Query 3' appears twice (for Persona 2 and Persona 3). The second should be Query 3 and the first Query 2, or the numbering should be sequential.","section":"Use Case: Querying the Climate Knowledge Graph"},{"comment":"The reference list contains placeholders that should not appear in a submitted manuscript: [41]–[45] and [50] read 'Author, A. A. (Year). Title of More Reference...' and must be completed or removed. Several entries also contain editorial notes such as '(Example, actual citation details may vary...)' and '(Note: This seems to be a duplicate or very similar...)' which need to be cleaned.","section":"References"},{"comment":"Figures 1 and 2 (ChatGPT vs. ClimatePub4KG graphs) are referenced but not visible in the provided text. Please ensure the figures are included with clear captions, and ideally show the actual output of the Cypher queries rather than schematic diagrams.","section":"Figures"},{"comment":"The paper states 'This paper is not about how we built the ClimatePub4KG but rather about how it can now be used for domain-specific search and climate information retrieval.' This is fine, but the use-case demonstration requires actual retrieval results. If the paper is intended as a vision/position statement, that should be stated explicitly, and the central claim should be softened accordingly.","section":"General"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be an early draft: the reference list contains placeholder entries and editorial notes, the query numbering is inconsistent, and the evaluation section is explicitly marked as future work. The central contribution—a working semantic retrieval backend—is not demonstrated in the paper. Even if the underlying KG is valuable, this submission does not provide the evidence needed for a research paper; it reads more as a project proposal or workshop position paper. If the authors add end-to-end query results, extraction-quality figures, and a rigorous comparison to baselines, a future submission on this topic could be appropriate for an IR or NLP venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, what you need to know: this is an extended abstract for the authors' prior ClimatePub4KG construction work, not a self-contained research paper. The abstract promises that the KG supports structured semantic queries that yield precise connections, but the manuscript shows only three hand-written Cypher listings, a schematic figure, and an anecdotal comparison to ChatGPT-4o. No query output, no precision/recall, no user study, no released graph or code. The conclusion even admits that benchmark evaluation is future work. So the reader's REJECT is on target.\n\nWhat is genuinely useful: the three use cases (cold-air outbreaks over North America, CMIP5 + NAO + Southeast US, PNA over US) are sensible examples of the kind of multi-hop retrieval that a domain KG should enable. And the underlying construction pipeline (ClimateIE, SciER, taxonomy-driven KG) is real prior work with actual publications. The paper's framing of KGs as a grounding layer for RAG is standard but reasonable.\n\nThe soft spots are substantial. Stripped of the use-case framing, this is a position statement. The central claim—that ClimatePub4KG enables precise discovery—rests entirely on the extraction quality of ClimateIE and SciER, but no accuracy numbers are given anywhere in this manuscript. The one comparison with ChatGPT-4o is not quantified; we are told the graph is 'richer' and 'more accurate' based on a single example, and the figures don't even include the output graphs (they appear as empty boxes in the text). The reference list contains placeholders '[41]–[45], [50]' with generic author names, and duplicate entries (e.g., [29]/[39], [30]/[40]). Typos like 'were present', 'Cipher', 'Devloper' signal a draft state.\n\nThat said, the stress-test note's worry about silent empty results is correct: queries like Listing 2 depend on 'TargetsLocation' edges and Wikidata descriptions that may not exist in the graph; without a spot-checked query result, the Cypher is plausibly correct but unverified.\n\nBottom line: this deserves to be a workshop discussion paper at most. A serious editor should not send this to full peer review in its current form; it needs either real evaluation or an explicit reframing as a vision/position statement. For researchers working on scientific KG retrieval, the use cases are worth seeing, but wait for the version with actual results.","headline":"A use-case sketch for the authors' existing ClimatePub4KG, with no query results, no evaluation, and placeholder references; the central retrieval claim is asserted but not demonstrated.","tokens_in":9600,"tokens_out":2189,"would_cite":false,"duration_ms":21216,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A climate knowledge graph answers research queries that keyword search cannot, by retrieving papers through typed relations among models, regions, datasets, and weather phenomena.","keywords":["climate knowledge graph","semantic retrieval","Cypher queries","climate models","teleconnection patterns","retrieval-augmented generation","scientific literature search","entity-relation extraction"],"falsifier":"Run the three example Cypher queries against the actual ClimatePub4KG and have a domain expert judge every returned paper: does it really mention, say, a CMIP5 model, the North Atlantic Oscillation, and the southeastern U.S.? If a large fraction are false positives, the semantic-retrieval claim collapses. Also compare against plain full-text keyword search on the same corpus: if a simple keyword query returns the same relevant papers, the graph is not adding retrieval value.","tokens_in":8822,"feed_emoji":"🌐","tokens_out":5522,"duration_ms":53886,"temperature":0.7,"pith_summary":"This paper argues that a domain-specific knowledge graph built from climate publications—ClimatePub4KG—enables a form of literature retrieval that keyword search cannot: structured queries that combine models, datasets, regions, teleconnection patterns, and weather events into single questions. The authors demonstrate with three worked Cypher queries that researchers can ask, for example, which papers mention CMIP5 models and the North Atlantic Oscillation in the southeastern United States, and get back specific paper titles with the matching sentences. They further claim the graph can serve as a grounding backend for retrieval-augmented generation, giving large language models a verifiable source of climate facts. A sympathetic reader would care because this points toward faster literature reviews and more transparent AI answers in a field where misremembered model names or region mismatches can mislead.","feed_headline":"Climate knowledge graph answers queries keywords can't","feed_subtitle":"Cypher queries over climate publications find precise links between models, regions, and phenomena—and can ground AI answers.","key_machinery":"The load-bearing object is the entity–relation schema of ClimatePub4KG: typed nodes for climate models, datasets, locations, weather events, variables, and teleconnection patterns, connected by typed edges such as Mention and TargetsLocation. The queries are written in Cypher, the graph query language of Neo4j, which matches graph patterns; for instance, one query asks for papers whose Mention sentences contain 'CAOs' or 'WW' and whose weather event node targets NORTH_AMERICA. This pattern-matching over explicit relations is what converts a literature search into a structured knowledge-discovery operation, and it is the same machinery that could later be driven by an LLM translating natural","core_discovery":"The paper's central claim is that ClimatePub4KG, populated by the earlier ClimateIE and SciER pipelines, can answer precise, multi-hop scientific questions by graph traversal rather than keyword matching. The worked examples show the query pattern: a paper node is connected to nodes for models, teleconnection patterns, locations, and weather events through typed edges like Mention and TargetsLocation, and a Cypher query filters on each dimension at once. The paper also reports a direct comparison: for the same natural-language question, a general-purpose conversational AI produced a graph with wrong relation directions (e.g., CMIP models 'influencing' rainfall instead of being evaluated agai","pith_inferences":["If extraction accuracy is not high (the paper gives no precision/recall numbers), a perfect Cypher query can still return irrelevant or missing papers; the example results should be read as illustrative until extraction quality is measured.","The paper flags model genealogy—shared code among CMIP models—as a key challenge but does not claim to solve it; without explicit lineage or similarity edges, ensemble-bias questions like 'which models are truly independent' cannot be answered from the graph as presented.","A natural-language-to-Cypher interface would widen the user base well beyond programmers; the paper says it is in progress but shows no working demo.","The reference list contains several placeholder entries ('Author, A. A.'), so parts of the related-work scaffolding are not verifiable in this version; the argument itself does not depend on them."],"forward_implications":["Researchers can combine several dimensions in a single query—model generation, teleconnection pattern, region—and return only papers that mention all of them, with the supporting sentence.","The graph can act as a factual grounding source for LLM-based question answering, so answers can be traced to specific publications rather than generated from parametric memory.","Systematic reviews can be partially automated by programmatically selecting studies by structured criteria.","The comparison with an unaided chatbot suggests that KG-backed retrieval is more reliable for directional relations and for providing citations.","Since the KG combines Wikidata descriptions with entity names, queries can handle synonyms and variant phrasings more robustly than exact keyword matching."],"fun_headline_variants":["Climate papers become a graph for precise scientific queries","Semantic search over climate literature finds hidden links","Ask climate questions the KG way, not keyword way","Climate knowledge graph: multi-hop answers from publications","Beyond keywords: climate graph queries ground AI answers"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central claim depends on the reliability of the entity and relation extraction (ClimateIE, SciER) that populates the graph; the paper reports no precision or recall figures, so a correct query can still return wrong or incomplete results.","fun_headline_variants_meta":{"raw":{"variants":["Climate papers become a graph for precise scientific queries","Semantic search over climate literature finds hidden links","Ask climate questions the KG way, not keyword way","Climate knowledge graph: multi-hop answers from publications","Beyond keywords: climate graph queries ground AI answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000131,"raw_usage":{"total_tokens":920,"prompt_tokens":653,"completion_tokens":267,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":397,"completion_tokens_details":{"reasoning_tokens":195}},"tokens_in":397,"tokens_out":267,"duration_ms":3685,"temperature":1.0,"reasoning_tokens":195,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T18:10:33.597813+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the three example Cypher queries against the actual ClimatePub4KG and have a domain expert judge every returned paper: does it really mention, say, a CMIP5 model, the North Atlantic Oscillation, and the southeastern U.S.? If a large fraction are false positives, the semantic-retrieval claim collapses. Also compare against plain full-text keyword search on the same corpus: if a simple keyword query returns the same relevant papers, the graph is not adding retrieval value.","supporting_citations":[],"review_version":1}