{"id":"9604b96f-08a2-46f0-82f5-76956a1582bc","arxiv_id":"2608.07954","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Caching one-hop knowledge-graph neighborhoods with LRU or LFU prevents repeated graph queries in KGQA systems, giving up to 1.91x faster graph retrieval but only about 1.06x end-to-end speedup.","lead":"This paper introduces KGCache, an in-memory cache that stores the one-hop neighborhood of knowledge-graph entities so that repeated lookups in LLM question-answering pipelines can be served without hitting the database again. It shows that WebQSP and CWQ queries reuse many of the same entities, that graph retrieval becomes up to 1.91x faster, and that the end-to-end speedup is small because graph lookups are only a minor part of the full pipeline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline KG speedups are measured on dataset-specific subgraphs whose per-request latency profile may not transfer to a real KG backend; if the request-cost distribution changes, the 1.35x/1.91x numbers are not established outside the evaluation harness.","rationale":"The paper's primary contribution is the entity-level cache, and its internal evidence (hit rates, policy comparisons, explicit non-upper-bound labeling of the Oracle) is coherent. The semantic-cache F1 issue is real but explicitly flagged by the authors and requires only repeated trials and a CWQ check; it is secondary to the entity-cache claim. The most load-bearing condition for the central claim is that the subgraph backends used for latency measurement produce request-cost profiles representative of a real KG backend. This is not an internal inconsistency, but it is the point where the claimed speedups are least secure when generalized. A single reproduction on a larger or full-backend load would settle whether the concern lands. Since the reader already issued a CONDITIONAL verdict that accounts for this external-validity risk, no verdict change is warranted.","tokens_in":11567,"tokens_out":12167,"duration_ms":129112,"concrete_test":"Reproduce the Table 2 runtime experiment on a backend loaded with the full Freebase dump (or, if impractical, the 2-hop closure of all question entities instead of only triples incident to question-associated identifiers), using the same 400-question ToG traces and LRU/LFU at capacity 1000. Compare the resulting KG speedups and per-request miss-latency distributions against the reported 1.35x/1.91x. If the speedup changes by more than about 20%, the subgraph-specific measurement is confirmed as a load-bearing limitation; if it remains within noise, the transferability concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 constructs a dataset-specific collection of incident triples (retaining every triple whose subject or object references a question-associated Freebase identifier) rather than a full or vertex-induced subgraph, and explicitly states that reported KG latency should be interpreted for these evaluation-specific subgraphs. Table 2 then reports no-cache KG times of only 57.0-57.6 s for 400 questions. The speedup from caching is not determined by hit rate alone: WebQSP and CWQ have similar hit rates (34.6-38.6%) but materially different speedups (1.35x vs 1.91x), showing sensitivity to the request-cost distribution. On a production backend, SPARQL planning cost, network transfer, and neighborhood sizes differ, so the saved-time fraction and the headline 'up to 1.91x' are not established outside the subgraph. The paper's caveat is honest, but the abstract's central claim omits it. This is the least secure condition in the causal chain from observed entity reuse to the claimed KG-retrieval acceleration.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes KGCache, an in-memory cache that stores one-hop knowledge-graph neighborhoods at entity granularity and is placed between a KGQA reasoning engine and a KG backend. The authors characterize entity reuse in WebQSP and CWQ, evaluate LRU, LFU, and a trace-aware Oracle policy on cache hit rates and on KG retrieval time, and explore a semantic-context cache that reuses retrieved KG context for similar questions. The main reported results are entity-reuse fractions of 62.0% (WebQSP) and 67.0% (CWQ) of traversal-level entity occurrences, entity-cache KG speedups up to 1.91x (LFU on CWQ), and semantic-cache full-system speedups up to 1.06x on WebQSP.","tokens_in":11759,"tokens_out":9057,"duration_ms":88605,"significance":"The workload characterization is a useful and falsifiable empirical finding, and the entity cache is a simple, orthogonal optimization that leaves the LLM call sequence unchanged and could benefit any KGQA system exhibiting repeated entity access. The paper is unusually honest about its limitations: it explicitly labels the Oracle as not a theoretical upper bound, states that measured latencies apply to evaluation-specific subgraphs, and acknowledges the absence of repeated trials for the semantic F1 experiments. If the speedups transfer to a production KG backend, the contribution is practically relevant; however, transferability of the measured latency profile is the main open risk.","major_comments":[{"comment":"The abstract states that \"Entity caching accelerates KG retrieval by up to 1.91x,\" but the runtime experiments in Table 2 are run on dataset-specific incident-triple subgraphs, and Section 4 explicitly says the reported KG latency \"should be interpreted for these evaluation-specific subgraphs.\" Since Table 2 shows that similar hit rates yield very different speedups (WebQSP 34.8% hit rate gives 1.35x, CWQ 38.6% gives 1.91x), the speedup depends on the request-cost distribution, which may differ on a full production backend. The paper should temper the abstract or provide evidence that the cost distribution transfers.","section":"Section 4 / Abstract"},{"comment":"The text in Section 5.4 says \"Oracle is excluded because it requires the complete request trace in advance,\" yet Table 2 lists Oracle rows with concrete KG times and speedups (2.08x and 3.57x). This is a direct contradiction. The authors must clarify whether the Oracle times were measured after pre-population, simulated, or estimated from hit rates; if Oracle was not actually run, the rows should not appear as if they came from the same runtime experiment.","section":"Section 5.4 / Table 2"},{"comment":"The semantic-cache F1 results at tau=0.90 are based on a single run, and the paper itself states that \"Without repeated trials, we cannot distinguish a cache effect from LLM nondeterminism\" and that CWQ remains untested. This transparency is good, but Section 7 then claims \"We proved that both work on both iterative and one-shot planning approaches,\" which overstates the evidence: the entity cache is only empirically evaluated with ToG, and the semantic cache's quality safety is explicitly unresolved. The conclusion should be reworded to match the stated evidence.","section":"Section 5.5 / Section 7"}],"minor_comments":[{"comment":"The sentence \"75.9% and 44.7% of entities recur across CWQ and WebQSP benchmarks\" is imprecise; Figure 1a reports percentages of entity mentions (occurrences) that are reused, not percentages of distinct entities.","section":"Section 1"},{"comment":"Table 3 does not state the similarity threshold used for the semantic hits; specify whether all rows use tau=0.90 as in Figure 5b.","section":"Table 3"},{"comment":"The \"per-hit speedup\" shown in Figure 5a is not defined in the text; please provide the formula or a prose definition.","section":"Section 5.5"},{"comment":"There are two typos: \"sysmptom\" should be \"symptom,\" and \"then a miss\" should be \"than a miss.\"","section":"Section 5.5"},{"comment":"The phrase \"We proved that both work\" is too strong for an empirical evaluation; consider \"we demonstrated\" or \"we found.\"","section":"Section 7"},{"comment":"Please clarify how the 400-input subset for each dataset was selected and whether the reported results are stable across different random subsets.","section":"Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper's explicit caveats are a strength, but the Oracle inconsistency in Table 2 is a factual issue that must be resolved before publication. The transferability concern is substantial; if the authors cannot run on a full backend, a cost-model analysis or an explicit statement that the speedups are subgraph-specific would be needed. The contribution is a useful systems-level optimization, so the fit with the journal/venue depends on the scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nKGCache is a modest, honest systems paper. The new work is the workload characterization and the placement of the cache: entity-reuse rates of 62-67% at traversal level on WebQSP and CWQ, and a semantic cache that reuses KG context rather than full LLM responses, which is a real distinction from Dominic et al. The evaluation is internally consistent: hit rates line up with the reported speedups, and the authors explicitly label the Oracle as not an upper bound. They also admit, in the conclusion, that KG retrieval is a small fraction of total runtime, which is why the end-to-end speedup is only 1.06x. That level of candor is rare.\n\nThe main soft spot is external validity. Section 4 constructs dataset-specific collections of incident triples and says the latency numbers should be interpreted for those subgraphs. The abstract's 'up to 1.91x' omits that caveat, and the stress-test note is right that hit rate alone doesn't determine speedup—WebQSP and CWQ have similar hit rates but different speedups, so the saved-time fraction depends on the request-cost distribution. On a production backend, those numbers may not transfer. That said, the paper flags the issue itself; it's an honest limitation, not a hidden one.\n\nThe semantic-cache section is thinner. It's only tested on WebQSP, with no repeated trials, and the paper itself says it can't distinguish a cache effect from LLM nondeterminism, and that CWQ remains untested. The F1 differences are within 1.7 points, so no large degradation is visible, but 'no large degradation' is not the same as 'safe.' The conclusion's 'we proved' overstates what is, at best, a well-executed empirical demo.\n\nWho is this for? Anyone building KGQA pipelines who wants numbers on how much repeated retrieval actually happens, and what an LRU/LFU cache buys. It's not a breakthrough, but it's a legitimate systems contribution with reproducible code and honest caveats. It deserves a serious referee, not a desk reject. I'd suggest asking for full-graph or more representative latency measurements, repeated trials for the semantic cache, a CWQ accuracy check, and a conclusion that says 'evaluated' instead of 'proved.'","headline":"Honest, modest systems paper with useful reuse data and real caveats on subgraph-only latency and semantic caching; deserves peer review with revision.","tokens_in":12308,"tokens_out":2745,"would_cite":true,"duration_ms":27476,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"KGCache places an in-memory cache for one-hop knowledge graph neighborhoods between the KGQA engine and the KG backend, cutting repeated backend queries and speeding up KG retrieval by up to 1.91x while preserving the LLM call sequence.","keywords":["knowledge graph question answering","entity caching","one-hop neighborhood","semantic caching","Think-on-Graph","Reasoning-on-Graph","LRU","LFU"],"falsifier":"Run the same WebQSP and CWQ traces against a full Freebase backend or another production KG instead of the dataset-specific incident-triple subgraphs, and repeat the semantic-cache F1 evaluation multiple times at $\\tau=0.90$; if the per-hit KG speedup or the F1 gap moves outside the reported ranges, the central caching claim needs qualification.","tokens_in":11368,"feed_emoji":"⚡","tokens_out":5141,"duration_ms":51540,"temperature":0.7,"pith_summary":"KGCache is a caching layer for knowledge-graph question answering that stores one-hop neighborhoods of entities between the reasoning engine and the KG backend. The paper's central claim is that repeated entity retrieval is a large, concentrated share of KGQA work: 62.0% of traversal-level entity occurrences in WebQSP and 67.0% in CWQ are reused, so an LRU or LFU cache can serve many requests from memory. In the evaluated setups this accelerates KG retrieval by up to 1.91x, reduces backend queries by up to 38%, and each cache hit is up to 3.73x faster than a miss. A complementary semantic-context cache that skips traversal for similar queries reaches up to 13.5% hit rate and up to 1.06x full-system speedup on WebQSP. A sympathetic reader would care because it offers a drop-in optimization that leaves the LLM call sequence unchanged and works across both iterative and one-shot KGQA paradigms.","feed_headline":"Caching KG neighborhoods speeds retrieval up to 1.91x","feed_subtitle":"Repeated entity lookups are served from memory, cutting KG retrieval time by up to 1.91x.","key_machinery":"The central object is an entity-granularity cache keyed by entity identifier $e$ that stores the complete one-hop neighborhood $N(e)$, including all incident subject-relation-object triples, and sits between the KGQA engine and the KG backend. On a request for $N(e)$, a hit returns the neighborhood from memory and a miss fetches it from the backend and inserts it under an LRU, LFU, or trace-aware Oracle eviction policy. A second, complementary mechanism is the semantic-context cache, which stores a query embedding together with the verbalized KG context and reuses that context when an incoming query's cosine similarity meets a threshold $\\tau$; embeddings are computed with all-MiniLM-L6-v2 and compared by dot product in 384 dimensions. Together these mechanisms amortize repeated subgraph retrieval across questions without altering the answer-generation LLM call.","core_discovery":"The paper claims that KGQA workloads are dominated by repeated entity access, concentrated in a relatively small set of frequently visited entities, and that placing a cache of one-hop entity neighborhoods between the KGQA engine and the KG backend removes a large share of the repeated backend work. Entity caching accelerates KG retrieval by up to 1.91x, while semantic-context caching achieves up to 1.06x full-system speedup in the evaluated WebQSP configurations, with each hit being up to 3.73x faster. The cache does not change the LLM call sequence for entity-level hits, because each traversal step still invokes the reasoning engine, whereas the semantic cache reuses an already constructed context and skips both backend invocation and the intermediate traversal steps. The paper also finds that the benefits are not artifacts of question ordering, since shuffled-access traces match sequential-access results.","pith_inferences":["Extension: The reported 1.06x full-system speedup is bounded by KG retrieval being a small fraction of total runtime; if a production backend has slower per-request latency, the end-to-end gain from the same hit rates would grow, and if the backend is faster, the gain would shrink or disappear.","Extension: The semantic cache's threshold $\\tau=0.90$ is not validated as F1-neutral on CWQ, and the paper reports incomplete entity overlap for CWQ at every threshold, so deploying semantic caching on CWQ would require per-dataset accuracy checks before trusting hit-rate gains.","Extension: Because the cache stores unpruned one-hop neighborhoods, it preserves all information needed for question-specific LLM pruning; this design could be extended to a hybrid cache that also memoizes pruned subgraphs per question type, though the paper does not explore that.","Extension: The workload reuse patterns suggest that caching could be coordinated with LLM pruning schedules, for example by prefetching neighborhoods of frequently reused entities before a traversal begins, a testable optimization not evaluated in the paper."],"forward_implications":["KGQA engines that use iterative traversal can cut a large fraction of backend KG queries with a small cache: LRU reaches a 30.7% hit rate on WebQSP with only 50 cached entities.","Entity caching accelerates KG retrieval by up to 1.91x on CWQ and 1.35x on WebQSP while leaving the LLM call sequence unchanged, so the optimization composes with any frontend LLM.","Semantic-context caching can skip entire traversal pipelines for similar questions, reaching up to 13.5% cold-cache hit rate on WebQSP and 1.06x full-system speedup in the evaluated configurations.","The workload characterization predicts that other KGQA benchmarks with similarly skewed entity access will show comparable cacheability, since reuse persists under shuffled question order.","Multi-hop neighborhood caching is a plausible next step, but it requires preserving reusable structure across different questions that start from the same entity."],"supporting_citations":[{"why":"Defines the ToG iterative traversal paradigm that repeatedly requests one-hop neighborhoods, the primary workload KGCache optimizes.","marker":"[3]"},{"why":"Defines the RoG one-shot planning-retrieval-reasoning paradigm that KGCache's semantic cache and entity cache are also designed to support.","marker":"[6]"},{"why":"Supplies the CWQ benchmark dataset whose entity and semantic reuse patterns drive the caching motivation.","marker":"[7]"},{"why":"Supplies the WebQSP benchmark dataset and the evaluation scripts used for accuracy and F1 measurements.","marker":"[8]"},{"why":"Identifies Freebase as the knowledge graph backend whose triples are filtered into the evaluation-specific subgraphs.","marker":"[14]"},{"why":"Provides the Virtuoso KG backend used in the runtime latency experiments.","marker":"[12]"},{"why":"Provides the Oxigraph KG backend used as an interchangeable alternative in the runtime experiments.","marker":"[13]"},{"why":"Establishes the prior semantic-cache approach that caches full LLM responses, which KGCache contrasts by caching retrieved KG context instead.","marker":"[11]"}],"fun_headline_variants":["KGCache speeds KG retrieval up to 1.91x","Caching KG neighborhoods yields 1.91x faster retrieval","Entity caching for KGQA: up to 1.91x faster","KGCache: repeated entity lookups served from memory"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported speedups assume that latencies measured on evaluation-specific Freebase subgraphs transfer to a real production KG backend, and that $\\tau=0.90$ semantic reuse preserves answer quality beyond the single WebQSP run without repeated trials.","fun_headline_variants_meta":{"raw":{"variants":["KGCache speeds KG retrieval up to 1.91x","Caching KG neighborhoods yields 1.91x faster retrieval","Entity caching for KGQA: up to 1.91x faster","KGCache: repeated entity lookups served from memory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000264,"raw_usage":{"total_tokens":1612,"prompt_tokens":961,"completion_tokens":651,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":578}},"tokens_in":577,"tokens_out":651,"duration_ms":8119,"temperature":1.0,"reasoning_tokens":578,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:36:20.607728+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same WebQSP and CWQ traces against a full Freebase backend or another production KG instead of the dataset-specific incident-triple subgraphs, and repeat the semantic-cache F1 evaluation multiple times at $\\tau=0.90$; if the per-hit KG speedup or the F1 gap moves outside the reported ranges, the central caching claim needs qualification.","supporting_citations":[{"cited_title":"Reasoning on graphs: Faithful and interpretable large language model reasoning","cited_arxiv_id":null,"evidence_quote":"Defines the RoG one-shot planning-retrieval-reasoning paradigm that KGCache's semantic cache and entity cache are also designed to support."},{"cited_title":"The web as a knowledge-base for answering complex questions","cited_arxiv_id":null,"evidence_quote":"Supplies the CWQ benchmark dataset whose entity and semantic reuse patterns drive the caching motivation."},{"cited_title":"The value of semantic parse labeling for knowledge base question answering","cited_arxiv_id":null,"evidence_quote":"Supplies the WebQSP benchmark dataset and the evaluation scripts used for accuracy and F1 measurements."},{"cited_title":"Freebase: a collaboratively created graph database for structuring human knowledge","cited_arxiv_id":null,"evidence_quote":"Identifies Freebase as the knowledge graph backend whose triples are filtered into the evaluation-specific subgraphs."},{"cited_title":"Virtuoso universal server.https://virtuoso.openlinksw.com","cited_arxiv_id":null,"evidence_quote":"Provides the Virtuoso KG backend used in the runtime latency experiments."},{"cited_title":"Oxigraph","cited_arxiv_id":null,"evidence_quote":"Provides the Oxigraph KG backend used as an interchangeable alternative in the runtime experiments."},{"cited_title":"Knowledge graph-enhanced semantic cache for low-latency and cost-effective inference in large language models","cited_arxiv_id":null,"evidence_quote":"Establishes the prior semantic-cache approach that caches full LLM responses, which KGCache contrasts by caching retrieved KG context instead."}],"review_version":1}