{"id":"b50da015-951e-4f49-bbb4-7137d58abdf7","arxiv_id":"2505.16849","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Offline walks over a knowledge graph, verbalized into text and retrieved by embedding similarity, let a single LLM call answer multi-hop questions competitively without any fine-tuning.","lead":"This paper introduces Walk&Retrieve, a knowledge-graph retrieval method that turns graph walks into plain text and feeds the most relevant ones to an LLM along with the question. A smart generalist might read it because it shows that a simple, training-free retrieval baseline can match or beat heavier graph-RAG systems on question answering benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Outgoing-only walks in §2.1 assume every answer is reachable by outward edges; MetaQA's head-initial queries hide this, so the general zero-shot KG-RAG claim is not established.","rationale":"The reader's conditional verdict is supported. I focused on the outgoing-edge assumption because it is the least-secure condition for the method's central claim as a general zero-shot KG-RAG framework. The paper's own MetaQA result is credible given the dataset construction, but the claimed adaptability to arbitrary or dynamically updated KGs is not established because the traversal never follows incoming edges. The proposed test would settle whether the method actually retrieves answers from the graph or relies on the LLM's parametric memory for cases where the answer is not reachable outward. Secondary issues (test-set hyperparameter selection, missing error bars, no KG baselines on CRAG) also justify a conditional verdict, but they do not single out a mechanism-level flaw. Therefore the verdict remains CONDITIONAL, and the reader's conditional assessment is unchanged.","tokens_in":11961,"tokens_out":13898,"duration_ms":112571,"concrete_test":"For each MetaQA test question, compute whether the gold answer is reachable from the query entity via a directed path using only edges as stored in the KG. If a non-trivial fraction (>5%) are unreachable yet Walk&Retrieve-BFS still answers correctly, the model is relying on parametric knowledge rather than walk retrieval, invalidating the retrieval attribution. Better: run Walk&Retrieve-BFS on WebQSP (or on a copy of MetaQA with reversed edges for a random half of the facts) without adding inverse relations; if Hits@1 collapses relative to the reported 67.99, the method's success depends on the edge-direction alignment found in MetaQA.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The traversal in §2.1 defines N(v) and transition probabilities using only triples (v,r,v') in E. Consequently, the corpus for an entity contains only facts reachable by outgoing edges. For the central claim to hold as a general zero-shot KG-RAG method, every answer-relevant fact must be reachable outward from the question entity. MetaQA is constructed from triples with the question entity as head, so this hidden assumption is satisfied. On a canonicalized KG that stores each fact in a single direction (e.g., Freebase/WebQSP), many answers are tails of triples and would never appear in any walk; the method would retrieve nothing and would fall back to 'I do not know' or parametric hallucination. The paper never adds inverse relations, never evaluates on a KG with canonical directions, and claims adaptability to dynamically updated KGs without mentioning this requirement. This is a correctness risk for the claimed generality, not an internal contradiction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Walk&Retrieve, a zero-shot KG-based RAG framework that builds an offline corpus by walking the knowledge graph from each entity (random walks or BFS walks), verbalizing the walks with an LLM into natural-language sentences, and embedding the resulting text. At inference, it embeds the query, retrieves the k most similar nodes and their k most relevant verbalized walks by cosine similarity, and prompts an off-the-shelf LLM with the query plus that context, using a single LLM call. The main experiments on MetaQA (1-, 2-, and 3-hop) and CRAG compare against LLM-only, Vanilla RAG, SubgraphRAG, and RetrieveRewriteAnswer, reporting Hits@1, accuracy, hallucination, missing, and latency. The authors report that Walk&Retrieve-BFS outperforms all baselines on MetaQA, has lower hallucination and missing rates, and is competitive on CRAG, with ablations over walk depth, number of walks, and backbone LLMs.","tokens_in":12185,"tokens_out":6258,"duration_ms":55271,"significance":"If the MetaQA numbers are reliable, the paper makes a useful and falsifiable point: a simple, training-free, walk-based retriever with LLM verbalization can outperform trained KG-RAG baselines on a standard benchmark while using one LLM call per query. The strengths of the submission include released code, the use of official implementations for the two strong KG-RAG baselines, and ablations across walk depth and backbone LLMs. The main limitations—outgoing-only traversal, no described validation split, and incomplete KG-RAG baselines on CRAG—do not invalidate the MetaQA comparison but materially restrict the generality claims as currently written.","major_comments":[{"comment":"The traverse definitions use only outgoing edges: the neighbor set N(v) is defined by triples (v,r,v') in E, Eq. (1) gives nonzero probability only to outgoing edges, and Eq. (2) builds BFS layers from outgoing edges. The paper never adds inverse relations or discusses edge orientation. On MetaQA, the question entity is the head of the relevant triples, so answer entities are reachable by outward walks and the hidden assumption is satisfied. On a canonicalized KG that stores each fact in a single direction (e.g., Freebase/WebQSP), many answers are the subjects rather than the objects of the query entity's triples, so a walk rooted at the query entity would never encounter them even though the fact is present in the graph; the method would then fall back to 'I do not know' or parametric hallucination. This is a correctness risk for the claimed generality of zero-shot KG-based RAG, not an internal contradiction. Please either add inverse-edge or bidirectional traversal, evaluate on a benchmark with canonical directions, or explicitly state and justify the orientation assumption in the method and abstract.","section":"§2.1, Eqs. (1)-(2)"},{"comment":"The hyperparameters k=3, walk depth 4 for MetaQA, depth 3 for CRAG, and n_w=60 appear to be selected through preliminary experiments on the test sets; no validation split is described. This creates a risk of test-set overfitting, especially because the depth values differ between the two benchmarks. Additionally, the random-walk variant is stochastic, but no variance or error bars are reported across random-walk seeds. Please specify the validation protocol, report mean and standard deviation over multiple seeds for the RW variant, and show sensitivity to k and n_w on held-out data rather than only on the test sets.","section":"§3 Implementation Details; §4 Results"},{"comment":"The two KG-RAG baselines SubgraphRAG and RetrieveRewriteAnswer are not evaluated on CRAG (footnote 6), so the CRAG rows only support a comparison against LLM-only and Vanilla RAG. The paper nonetheless uses the CRAG results to claim scalability to large KGs and robustness. The absence of the strongest baselines weakens that claim; please either soften the scalability conclusion or include at least one KG-RAG baseline on CRAG, for example on a subsample of the CRAG KG or with a reduced configuration.","section":"§4, Table 2 (CRAG block)"}],"minor_comments":[{"comment":"The sentence 'Table 1 summarizes the QA performance of Walk&Retrieve and the baselines' should refer to Table 2; Table 1 reports dataset statistics.","section":"§4, first paragraph"},{"comment":"In the left-graph discussion, 'due to is systematic graph exploration' should read 'due to its systematic graph exploration'.","section":"§4, Fig. 4 discussion"},{"comment":"The notation C_RW = ∪ W_l suggests a set, but the ablation footnote reports that each node yields 60 duplicated and 8.74 unique random walks; please clarify whether the corpus retains duplicates as a multiset or deduplicates before indexing, since this affects both retrieval scores and the reported counts.","section":"§2.1"},{"comment":"The evaluation uses Llama-3.1-70B as both the generator and one of the two judges. The two-judge design mitigates self-preference, but the paper should report the agreement rate between the two evaluators (e.g., Cohen's kappa or the percentage of concordant labels) so the average scores are interpretable.","section":"§3, Evaluation Metrics"},{"comment":"The text states that values of n_w other than 60 'perform comparably,' but no numerical results are shown; please include the n_w ablation curve or a small table so the claim is verifiable.","section":"§4, Ablation of Walk Approach"}],"recommendation":"major_revision","confidential_remarks":"The MetaQA result is interesting and the method is simple enough to be a useful baseline. The major issue is the gap between the paper's broad zero-shot KG-RAG claim and the outgoing-only traversal assumption, which is satisfied by MetaQA's construction but not by KGs with canonical edge directions. A small additional experiment on WebQSP or a bidirectional-edge variant, plus a proper validation protocol, would make the claim defensible. The CRAG comparison is currently too incomplete to support the scalability conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Briefing: this is a workshop-scale empirical paper that recombines standard pieces—DeepWalk-style random walks, BFS layers in the spirit of RDF2Vec, LLM verbalization, cosine embedding retrieval—into a single offline-corpus, single-LLM-call zero-shot RAG pipeline. The combination is not in the prior literature, and the paper's main empirical claim holds up on its chosen benchmark: Walk&Retrieve-BFS beats SubgraphRAG and RetrieveRewriteAnswer on MetaQA in accuracy and truthfulness. I believe the result. The method is simple, the ablations make sense (BFS clearly better than RW), and the code is released. Using official baseline implementations and two backbone LLMs helps.\n\nNow the soft spots. The most important is the directionality assumption. Equations (1) and (2) define neighbors using only outgoing edges (v, r, v') in E. That means the corpus for an entity contains only facts reachable by walking outward. For the method to work as a general zero-shot KG-RAG system, every answer must be reachable outward from the query entity. MetaQA satisfies this because its questions start from the head of the triple path. A canonicalized KG that stores each fact in one direction—Freebase, for example—would make many answers unreachable (they are tails), and retrieval would come back empty. The paper claims adaptability to dynamic KGs and general zero-shot RAG without ever mentioning inverse relations or testing on such a graph. That is a genuine limitation of the headline claim, not an internal contradiction.\n\nSecond, the hyperparameters k=3, walk depth 4/3, and n_w=60 appear to be chosen on the test sets; there is no validation split and no variance across random-walk seeds. The gap over SubgraphRAG might be smaller under a cleaner protocol. This is a moderate concern, not a fatal one.\n\nThird, the CRAG results only compare against LLM-only and Vanilla RAG. The two KG-RAG baselines could not run at that scale, so 'outperforms on CRAG' actually means 'beats no-retrieval and triple-sampling RAG.' That's weak but honestly reported. The evaluator overlap (Llama-3.1 as generator and one of two judges) is partially mitigated by the GPT-4o judge.\n\nBottom line: this paper deserves a serious referee. The directionality issue should be the first request in revision—an inverse-edge variant and at least one canonical rather than head-initial KG would test the actual boundary. For practitioners it is a useful cheap baseline and for the field it sharpens the question of what graph traversal really buys in RAG. I would engage with it.","headline":"A clean zero-shot KG-RAG baseline with credible MetaQA numbers, but the outgoing-only walk traversal caps the generality claim.","tokens_in":12687,"tokens_out":3228,"would_cite":true,"duration_ms":26426,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Walk&Retrieve, a zero-shot framework that converts knowledge-graph walks into natural-language context, outperforms trained KG-based retrieval-augmented generation systems on standard QA benchmarks while using a single LLM call per query.","keywords":["knowledge graph RAG","graph walks","zero-shot retrieval","retrieval-augmented generation","question answering","random walks","BFS walks","hallucination reduction"],"falsifier":"Construct or use a QA benchmark over a knowledge graph that has been canonicalized so that the correct answer entity appears as the subject (not the object) of the triple connecting it to the query entity. Walk&Retrieve's traversal, which follows only outgoing edges $(v,r,v')$, should miss those triples and its Hits@1 should collapse; if it does not, then the method must be obtaining the facts from some source other than the walk corpus (for example, from the LLM's parametric memory).","tokens_in":11792,"feed_emoji":"🚶","tokens_out":5360,"duration_ms":39230,"temperature":0.7,"pith_summary":"The paper claims that retrieval-augmented generation over knowledge graphs does not need fine-tuned retrievers or multi-step LLM planning. It introduces Walk&Retrieve, a zero-shot pipeline that pre-computes, for each entity, a set of graph walks (random or breadth-first), verbalizes those walks into natural-language sentences with an off-the-shelf LLM, and indexes them. At query time, it embeds the question, retrieves the most similar entities and their most similar walks, and asks the same LLM to answer from that context in one call. On MetaQA, the BFS variant reaches 67.99% Hits@1 versus 43.88% for a trained subgraph-retrieval baseline, and it also reduces hallucination and missing-response rates while running faster than other RAG systems. The authors present walk-based corpus generation as a strong, scalable zero-shot baseline for KG-based RAG.","feed_headline":"Graph walks outperform trained KG-RAG, zero-shot","feed_subtitle":"One LLM call per query, no fine-tuning, and fewer hallucinations on QA benchmarks.","key_machinery":"The load-bearing mechanism is the walk-based corpus: for every vertex $v$ in the knowledge graph $G=(V,E,R)$, the method generates $n_w$ random walks of length $l$ (or a BFS spanning-tree partition into layers $L_i$ by shortest-path distance), verbalizes each walk into a natural-language sentence by prompting an LLM, and indexes both node-level and walk-level embeddings. At inference, the query is embedded with the same LLM, the top-$k$ nodes and their top-$k$ walks are selected by cosine similarity, and the answer is generated in a single LLM call with a prompt that instructs the model to say 'I do not know the answer' if the context is insufficient. The BFS variant avoids duplicate walks and explores systematically, which the paper attributes to its consistently higher truthfulness than the random-walk variant.","core_discovery":"The central claim is that a simple, fully zero-shot retrieval strategy — pre-computing bounded graph walks from every entity, verbalizing them into sentences, and retrieving the top-k most similar walks by embedding cosine similarity — is competitive with, and often better than, KG-RAG systems that fine-tune retrievers or generators. Concretely, Walk&Retrieve-BFS outperforms all baselines on MetaQA in answer accuracy and Hits@1, achieving 67.99% Hits@1 against 43.88% for SubgraphRAG, and yields the lowest hallucination and missing-response rates among KG-based RAG methods. The paper also reports that the method is robust across backbone LLMs (Llama-3.1, Mixtral, GPT-4o), scales to a larger KG (CRAG) where trained baselines cannot run, and adapts to dynamic KGs by regenerating only the walks of changed elements. The intended upshot is that the complicated machinery of learned graph retrievers and multi-step LLM reasoning may be unnecessary for KG-grounded QA.","pith_inferences":["If the empirical edge holds, a larger implication is that the corpus itself — not the retriever — is the limiting factor for KG-RAG quality, so research should focus on what to extract and how to verbalize it rather than on learned retrieval architectures.","The reliance on outgoing edges suggests that canonicalization of KG directionality could significantly affect results; augmenting walks with incoming edges (bidirectional traversal) might be a cheap, testable improvement that the paper does not explore.","The verbalization step's quality likely determines the ceiling: an LLM that verbalizes walks more coherently (e.g., with entity type information or relation paraphrases) may push accuracy higher without changing traversal.","The method's reported latency advantage should be re-examined on KGs where the pre-computation cost (BFS over the whole graph) dominates; for very large dynamic KGs, random walks with deduplication may behave differently than the paper's fixed setting."],"forward_implications":["KG-RAG can be a lightweight, zero-shot service: no retraining when the KG changes, only incremental walk recomputation for affected entities.","A single LLM call per query cuts inference latency compared with systems that interleave LLM calls with graph traversal steps.","Any off-the-shelf LLM can be used as the backbone, so improvements in LLM quality transfer directly to KGQA performance.","Walk-based retrieval provides a strong and simple baseline that future KG-RAG methods should be measured against."],"supporting_citations":[{"why":"Primary trained KG-RAG baseline (SubgraphRAG) that Walk&Retrieve outperforms on MetaQA.","marker":"[25]"},{"why":"Trained KG-RAG baseline (RetrieveRewriteAnswer) requiring fine-tuning, used for comparison.","marker":"[53]"},{"why":"Provides the MetaQA benchmark and its 1-, 2-, and 3-hop question subsets.","marker":"[58]"},{"why":"Provides the CRAG benchmark and the three-way accuracy/hallucination/missing evaluation methodology.","marker":"[55]"},{"why":"Supplies the BFS-layer graph traversal approach adapted for walk-based corpus generation.","marker":"[41]"},{"why":"Supplies the random-walk formalism and transition probabilities used for walk generation.","marker":"[38]"},{"why":"Defines the Vanilla RAG baseline that samples 1-hop triples, against which Walk&Retrieve is compared.","marker":"[45]"}],"fun_headline_variants":["Zero-shot walks beat fine-tuned KG-RAG on QA","Walk&Retrieve: zero-shot walks, no tuning, better QA","Graph walks: zero-shot RAG that outperforms trained","Forget fine-tuning: walks give better KG-RAG","Simple walk-based RAG beats trained baselines, zero-shot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that every fact needed to answer a question is reachable by walking only along outgoing edges from the query entity, so if the knowledge graph stores a fact in reverse direction (the answer is the subject of the triple), the walk will never retrieve it even though the fact exists.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot walks beat fine-tuned KG-RAG on QA","Walk&Retrieve: zero-shot walks, no tuning, better QA","Graph walks: zero-shot RAG that outperforms trained","Forget fine-tuning: walks give better KG-RAG","Simple walk-based RAG beats trained baselines, zero-shot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001029,"raw_usage":{"total_tokens":4345,"prompt_tokens":965,"completion_tokens":3380,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":3294}},"tokens_in":581,"tokens_out":3380,"duration_ms":19460,"temperature":1.0,"reasoning_tokens":3294,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:53:29.440762+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct or use a QA benchmark over a knowledge graph that has been canonicalized so that the correct answer entity appears as the subject (not the object) of the triple connecting it to the query entity. Walk&Retrieve's traversal, which follows only outgoing edges $(v,r,v')$, should miss those triples and its Hits@1 should collapse; if it does not, then the method must be obtaining the facts from some source other than the walk corpus (for example, from the LLM's parametric memory).","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Primary trained KG-RAG baseline (SubgraphRAG) that Walk&Retrieve outperforms on MetaQA."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MetaQA benchmark and its 1-, 2-, and 3-hop question subsets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the BFS-layer graph traversal approach adapted for walk-based corpus generation."}],"review_version":1}