{"id":"21d68e5f-ae20-4018-b3c7-d720b5f33119","arxiv_id":"2505.00443","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A distributed RAG framework using topic-aware random walk routing lets edge devices retrieve knowledge from peers with near-centralized accuracy and about half the messages of flooding.","lead":"This paper proposes a peer-to-peer version of retrieval-augmented generation, where each device keeps its own knowledge base and queries nearby peers instead of a central server. It reports simulation results showing accuracy close to centralized retrieval while sending fewer messages than flooding the network.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim depends on an unstated, potentially favorable knowledge-distribution split; until the split is specified and varied, the reported hit rates and message savings are not established.","rationale":"I read the paper as a systems proposal with a concrete algorithm, a legible evaluation, and a sensitivity analysis that credibly shows TARW is more stable than random-walk retrieval as the network grows. Those are real strengths. The load-bearing concern, however, is the same one the reader identified: the experimental protocol never specifies how knowledge is distributed across peers or how queries are assigned, and both dependent variables in the central claim depend directly on that distribution. If the split is favorable, TARW's high hit rate and message savings are inflated; if the split is unfavorable, the method may lose its advantage over flooding. This is a falsifiable and fixable missing condition rather than a demonstrated failure, so I do not think the paper should be rejected outright. The reader's CONDITIONAL verdict already captures this appropriately, and my stress-test does not move it. I would add the quantitative caveat that 'half as many messages' is not supported at the default 20-peer configuration in Table 1, where the reduction is much smaller, and that the abstract's unqualified wording should be tightened regardless of the split issue.","tokens_in":14100,"tokens_out":5040,"duration_ms":58684,"concrete_test":"Release the exact split and query-assignment procedure, then rerun the default configuration from Section 4.1 under three controlled conditions: (a) each knowledge snippet stored on exactly one peer, assigned uniformly at random; (b) duplication factor r=4, with each snippet replicated on four random peers; and (c) an anti-correlated split where the querying peer is selected from peers whose local knowledge base lacks the answer snippet. Report hit rate, F1, and average messages for TARW and flooding across at least 10 random seeds. If TARW's hit rate or message advantage degrades materially when r=1 or under the anti-correlated assignment, the headline claim should be restated as conditional on knowledge redundancy and locality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest load-bearing step is the unstated knowledge-distribution model behind Table 1 and Figure 3. Section 4.1 specifies the network generator (Barabási-Albert), H_max=6, k=4, and θ=0.8, and names the three public datasets, but it never says how each dataset is partitioned into per-peer knowledge bases, how queries and ground-truth answers are assigned to peers, how many peers hold a given answer snippet, or which embedding model implements the relevance threshold. TARW's advantage over flooding is precisely its ability to stop after a few topic-guided hops (Algorithm 1, Lines 10-17); if relevant snippets are replicated across many peers, or if the querying peer is usually close to a peer holding the answer, then almost any directed search will look successful. The reported MMLU hit rate of 98.11% with an average of 1.72 hops and 6.87 messages is compatible with a favorable split and does not by itself establish the abstract's claim that TARW uses half as many messages as flooding. At the default 20-peer setting, the measured reductions are about 37% on MMLU, 9% on Medical, and 29% on News; the roughly 50% reduction appears only at 100 peers in Figure 3. The central claim should therefore be read as conditional on an unquantified distribution of knowledge across peers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Distributed Retrieval-Augmented Generation (DRAG), a framework that removes the centralized knowledge base from RAG by storing knowledge across peer devices and routing queries with a Topic-Aware Random Walk (TARW). TARW extracts query topics with a local LLM, uses a relevance threshold to select knowledgeable peers, and maintains a cache of peer expertise. The authors compare DRAG-TARW against centralized RAG, random walk, and flooding baselines on MMLU, Medical Extended, and News datasets, reporting near-centralized accuracy with reduced message counts, scaling behavior of message overhead, and sensitivity to network size, connectivity, LLM choice, and the neighbor-selection parameter.","tokens_in":14428,"tokens_out":2502,"duration_ms":26521,"significance":"If the empirical claims hold, DRAG would be a useful step toward privacy-preserving and scalable RAG on edge networks, and the TARW routing idea is simple and plausibly transferable. The manuscript's strengths include a clearly formulated algorithm, open-source code, evaluation across three datasets, three open-weight LLMs, and multiple ablation experiments. However, the core quantitative claim depends on an underspecified knowledge-distribution setup, and the reported results lack statistical variation, so the current evidence is conditional rather than conclusive.","major_comments":[{"comment":"The paper never specifies how each dataset is partitioned into per-peer knowledge bases, how queries and ground-truth pairs are assigned to peers, how many peers hold a given answer snippet, or which embedding/relevance model implements Relevance(). The reported hit rates (e.g., 98.11% on MMLU) and message counts assume that relevant knowledge is reachable within H_max=6 hops; if relevant snippets are replicated across many peers or if querying peers are close to answer-holding peers, TARW's advantage over flooding is inflated. Because this distribution model is the load-bearing assumption behind Table 1 and Figure 3, the authors must specify the partitioning and replication strategy, then vary it (e.g., different duplication factors, adversarial query-to-peer assignments) to show the result is robust.","section":"§4.1 (Datasets), Algorithm 1 lines 10–13"},{"comment":"The abstract states that DRAG-TARW uses 'half as many messages as flooding.' At the default 20-peer setting in Table 1, the reductions over flooding are roughly 37% on MMLU (6.87 vs 10.91), 9% on Medical (8.82 vs 9.72), and 29% on News (7.82 vs 10.99); the approximately 50% reduction appears only at 100 peers in Figure 3. The abstract therefore overstates the headline result. The paper should either soften the claim to specify the scaling regime or present a rigorous scaling analysis showing when the 50% reduction holds.","section":"Abstract and §4.3 (Figure 3)"},{"comment":"All results are single-run point estimates without variance, confidence intervals, or significance tests. Given stochasticity in Barabási-Albert graph generation, query assignment, and LLM generation, the observed differences (e.g., EM 83.90% for DRAG-TARW vs 85.73% for CRAG on MMLU; message counts differing by less than a message on Medical) cannot be assessed for reliability. At minimum, the authors should run multiple seeds and report mean ± std over independently generated networks and knowledge assignments.","section":"Table 1, Figures 3–7"},{"comment":"The comparison against centralized RAG variants with incomplete knowledge (CRAG-0.7S, CRAG-0.5S, CRAG-0.7T, CRAG-0.5T) is not controlled. In those variants, the centralized knowledge base is deliberately reduced to 50–70% of snippets or topics, whereas DRAG has access to the full distributed knowledge across all peers. DRAG's superior F1 score in that comparison is therefore expected by construction and does not by itself demonstrate resilience to incomplete knowledge. To support the resilience claim, the authors must apply a comparable completeness constraint to the distributed setting (e.g., remove a fraction of topics or snippets from all peers) and show DRAG still outperforms the equally limited centralized baseline.","section":"Figure 2"}],"minor_comments":[{"comment":"The relevance function used in Line 10–11 is never defined beyond 'semantic similarity'; please state which embedding model (e.g., sentence-transformers or the local LLM itself) computes Relevance(K_i, q), and report the prompt/temperature used for topic extraction.","section":"§4.1, Algorithm 1"},{"comment":"The Medical Extended dataset is described as synthetic; please clarify what constitutes a peer's knowledge base for this dataset and how patient-related snippets are split among peers, since this affects the privacy interpretation.","section":"§4.1, Medical Extended dataset"},{"comment":"The paper uses placeholder publication venue and DOI fields in the ACM reference format; these should be completed or removed if the manuscript is intended for arXiv distribution.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this is a solid systems paper, not a breakthrough. It puts together decentralized RAG with a topic-aware random walk and an expertise cache in a way I haven't seen in the federated RAG or query-routing literature. The evaluation is broad enough to support the main qualitative claim: on three datasets, TARW lands within a few F1 points of centralized RAG and sends fewer messages than flooding. That is a real, useful result for edge applications that don't want a central knowledge base.\n\nWhat it does well: the algorithm is clearly described, the baselines are reasonable (CRAG, NoRAG, random walk, flooding), and the sensitivity analysis covers network size, connectivity, model choice, and the k parameter. Figure 4 shows a nice convergence effect from caching. The authors also discuss privacy and reliability limitations honestly in Section 3.3.\n\nThe soft spots are about evidence quality, not the core idea. The biggest one, which the stress-test flags correctly, is that the knowledge distribution across peers is never specified. We don't know how the datasets are partitioned, how queries and answers are assigned to peers, or how many peers hold a given snippet. With hit rates around 98% and average hops under 2 on a 20-peer network, the split must be quite favorable—relevant knowledge is probably replicated or close to the querying peer. If the split were adversarial or sparse, TARW's message savings would shrink and perhaps disappear. That assumption needs to be stated and varied experimentally. Second, there are no variance bars; all numbers look like single-run values. Third, the abstract's \"half as many messages\" claim is true only at 100 peers (about 50% on MMLU and News), not at the default 20 peers where the savings are 37%, 9%, and 29%. That's not fatal, but the headline overstates the default configuration. Fourth, the comparison in Figure 2 against intentionally incomplete CRAG variants is a bit stacked—of course a centralized system with half its knowledge does worse. It still makes a fair point about resilience, but it's not the strongest evidence for DRAG.\n\nNone of this breaks the paper. The central argument—that topic-guided routing can approximate centralized retrieval at lower communication cost in a distributed setting—holds up under the stated favorable split. The next revision should disclose the partitioning scheme, test at least one alternative split, and run multiple seeds.\n\nMy recommendation: send it to peer review. It deserves a serious referee, and with a revision that closes the knowledge-distribution gap, it would be a useful contribution to the distributed LLM literature.","headline":"A legitimate decentralized RAG proposal whose headline message savings are real but only demonstrated under a favorable, unstated knowledge distribution; deserving of peer review with revisions.","tokens_in":14927,"tokens_out":5016,"would_cite":true,"duration_ms":43083,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A topic-aware random walk lets a distributed RAG system match centralized accuracy at roughly half the communication cost.","keywords":["retrieval-augmented generation","distributed systems","peer-to-peer networks","topic-aware random walk","edge computing","large language models","knowledge retrieval","privacy-preserving AI"],"falsifier":"Re-run the MMLU, medical, and news experiments holding the network and parameters fixed while varying the per-peer knowledge split from high topic duplication to strict one-topic-per-peer silos. If TARW's hit rate and message savings weaken substantially in the silo condition, the near-centralized claim is an artifact of knowledge overlap rather than a property of topic-aware routing.","tokens_in":13898,"feed_emoji":"📡","tokens_out":8890,"duration_ms":84753,"temperature":0.7,"pith_summary":"The paper sets out to show that retrieval-augmented generation does not need a central knowledge base. It proposes a peer-to-peer framework, DRAG, in which each edge device keeps its own private knowledge base and LLM, and a Topic-Aware Random Walk (TARW) routes the query toward peers likely to hold the answer. Across MMLU, medical, and news benchmarks, TARW retrieves relevant knowledge for 96 to 98 percent of queries and lands F1 scores within roughly two to four points of centralized RAG, while sending fewer messages than flooding at 20 peers and close to half as many at 100 peers. A reader should care because the same architecture removes the central store, which is a privacy and scaling bottleneck, at a modest and partly tunable accuracy cost.","feed_headline":"Distributed RAG matches centralized accuracy at half the messages","feed_subtitle":"A topic-guided peer-to-peer walk keeps 96-98% of queries on target while flooding sends nearly twice the traffic.","key_machinery":"The object that carries the argument is the TARW algorithm (Algorithm 1), a bounded random walk whose steps are reweighted by topic. At the querying peer, the local LLM extracts topics from the query; each visited peer computes a local retrieval relevance score; if that score clears the threshold theta = 0.8, the peer's knowledge is privacy-filtered and returned, and the peer is recorded in an expertise cache under the query's topics. If no peer can answer, the current peer ranks unvisited neighbors by the cached topic-expertise match and forwards the query to the top k, with a max hop count of 6. The cache is the mechanism that turns raw exploration into learned routing: as successful queries accumulate, message counts converge downward, and the sensitivity experiments show the accuracy jump comes mainly from moving the neighbor-selection count from k=2 to k=4.","core_discovery":"The central claim is that a decentralized RAG system can come close to the accuracy of a centralized one if knowledge discovery is topic-guided rather than blind. The paper's experiments show DRAG-TARW reaching a 98.11 percent hit rate and 83.92 F1 on MMLU against 99.81 percent and 85.75 for centralized RAG; on medical data, 98.67 percent hit rate and 90.58 F1 against 99.81 percent and 94.46; on news, 96.86 percent and 74.73 against 99.16 percent and 76.66. Meanwhile the blind random-walk baseline retrieves the right knowledge only 21 to 28 percent of the time, and its F1 collapses as the network grows. The conclusion the paper draws is that the distributed setting itself does not cause the accuracy loss; the routing policy does, and a topic-aware policy with a learned peer-expertise cache keeps the loss small while cutting communication.","pith_inferences":["The privacy benefit is architectural: no central server holds all data, but shared snippets are not encrypted or anonymized by construction, so a realistic deployment would need to pair DRAG with differential privacy or traffic-obfuscation mechanisms.","Because the advantage rests on topic overlap among peers, the natural stress test is a silo split where each topic lives on exactly one peer; the paper's own k=2 result (F1 56.27 on MMLU) suggests the near-centralized performance may not transfer to disjoint expert networks.","The paper counts each visited peer as one message, so real deployments also pay for routing-table maintenance, cache freshness, and retries; the half-message saving is a lower bound on communication overhead.","Treating the LLM as the router is a transferable idea: topic extraction plus an expertise cache could be applied to hierarchical federated search or multi-hop retrieval over private document collections where the bottleneck is finding who holds what without a central index."],"forward_implications":["DRAG-TARW's F1 stays within 1.8 to 3.9 points of centralized RAG on the three benchmarks, while flooding achieves similar accuracy at notably higher message counts.","At 100 peers the communication saving approaches half: 27.67 messages versus 53.34 on MMLU and 23.71 versus 48.83 on News.","Retrieval without topic guidance fails in this setting: the random-walk baseline's hit rate is 21.96 to 27.62 percent and falls to 3.57 F1 on MMLU at 100 peers.","Against incomplete central knowledge bases, DRAG is more resilient: when the central base is cut to 70 percent of its samples, centralized F1 on MMLU drops to 61.54 percent while DRAG-TARW holds 83.92 percent.","The system gets cheaper over time: on MMLU with 100 peers, average messages per query fall from 34.73 at 500 queries to 27.67 at 10,000 as the peer-expertise cache fills."],"supporting_citations":[{"why":"Supplies the original RAG formulation that the centralized baseline (CRAG) instantiates and that DRAG extends to a peer-to-peer setting.","marker":"[14]"},{"why":"Supplies the scale-free network model used to generate the peer-to-peer topologies in the simulations.","marker":"[38]"},{"why":"Provides the RAG benchmarking perspective that shapes the experimental comparison across datasets and metrics.","marker":"[13]"},{"why":"Frames RAG as the standard remedy for LLM hallucination and defines the retrieval-plus-generation pipeline that DRAG adapts.","marker":"[6]"}],"fun_headline_variants":["Distributed RAG matches central accuracy at half the messages","Peer-to-peer RAG: topic-aware walk slashes traffic, holds accuracy","Decentralized retrieval beats blind flooding with topic guidance","No central KB: DRAG keeps accuracy while cutting messages in half"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that relevant knowledge for almost every query sits within a few hops of the peer asking; the paper never says how it splits the datasets into per-peer knowledge bases, so a favorable split that duplicates topics across many peers could make TARW look better than flooding than a disjoint split would.","fun_headline_variants_meta":{"raw":{"variants":["Distributed RAG matches central accuracy at half the messages","Peer-to-peer RAG: topic-aware walk slashes traffic, holds accuracy","Decentralized retrieval beats blind flooding with topic guidance","No central KB: DRAG keeps accuracy while cutting messages in half"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000145,"raw_usage":{"total_tokens":1190,"prompt_tokens":965,"completion_tokens":225,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":153}},"tokens_in":581,"tokens_out":225,"duration_ms":2931,"temperature":1.0,"reasoning_tokens":153,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:42:21.377469+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the MMLU, medical, and news experiments holding the network and parameters fixed while varying the per-peer knowledge split from high topic duplication to strict one-topic-per-peer silos. If TARW's hit rate and message savings weaken substantially in the silo condition, the near-centralized claim is an artifact of knowledge overlap rather than a property of topic-aware routing.","supporting_citations":[{"cited_title":"Retrieval-augmented generation for knowledge-intensive nlp tasks,","cited_arxiv_id":null,"evidence_quote":"Supplies the original RAG formulation that the centralized baseline (CRAG) instantiates and that DRAG extends to a peer-to-peer setting."},{"cited_title":"Error and attack tolerance of complex networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the scale-free network model used to generate the peer-to-peer topologies in the simulations."},{"cited_title":"Benchmarking large language models in retrieval-augmented generation,","cited_arxiv_id":null,"evidence_quote":"Provides the RAG benchmarking perspective that shapes the experimental comparison across datasets and metrics."}],"review_version":1}