{"id":"5b94e986-3d6a-4dd6-bbfd-ac58237b1744","arxiv_id":"2509.06444","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"HyFedRAG is a federated RAG framework over heterogeneous data with local anonymization and three-tier caching, but the experiments do not support its headline performance and privacy claims.","lead":"This paper proposes HyFedRAG, a federated retrieval-augmented generation system that lets hospitals with different data formats (SQL, knowledge graphs, text) query each other's data without sharing raw records. The reported performance gains rest on a text-only retriever with hyperparameters tuned on the test set, and the full federated pipeline is never compared against a centralized baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim unsupported: no end-to-end federated RAG experiment; generation and latency comparisons are absent.","rationale":"The reader's REJECT verdict is well-founded. The paper's reported retrieval gains (Table 1) come from a text-only hybrid retriever whose α is tuned on the evaluation set (Section 4.3), and Table 2 shows severe degradation for SQL and KG. More critically, there is no evaluation of the complete HyFedRAG system: no multi-client scenario, no server-side fusion of de-identified summaries, no generation-quality metric, and no centralized RAG baseline. The conclusion's claim of 'outperforming centralized baselines' is therefore unsupported by any direct experiment. The reader's weakest_assumption about semantic preservation in Section 3.4 is a related but distinct concern: even if the full pipeline were run, the anonymization could destroy utility. However, the primary load-bearing gap is the absence of any end-to-end test. This gap alone justifies rejection, regardless of whether the summarization preserves semantics. The proposed concrete test would directly settle the central claim by comparing the full federated pipeline to a centralized RAG on generation quality and latency. No ad hominem is intended; the critique targets the experimental evidence, not the authors' integrity.","tokens_in":10521,"tokens_out":4585,"duration_ms":47712,"concrete_test":"Build a three-client federated setup over a 10k-record subset of PMC-Patients (one TEXT client, one SQL client, one KG client). Run the complete pipeline on 500 held-out queries: local retrieval → local LLM summarization with Presidio redaction → server-side LLM fusion. Compare final answer quality (e.g., faithfulness, factual F1, or ROUGE) against a centralized RAG that accesses all raw data across the three formats. Also measure real end-to-end latency for both configurations. If the federated pipeline's answer quality falls more than 10% below the centralized baseline, or if the latency reduction is not reproduced, the claimed superiority fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that HyFedRAG outperforms centralized RAG in retrieval quality, generation consistency, and system efficiency—is not established because no experiment runs the full HyFedRAG pipeline. Section 4.1 defines three single-modality datasets (TEXT, SQL, KG) and Table 1 reports retrieval metrics for a text-only retriever against text-only baselines. Table 2 reports per-modality retrieval for the individual modules. There is no multi-client configuration where distinct clients hold different modalities, generate privacy-preserving summaries, and a server-side LLM fuses them into a final answer. Consequently, 'generation consistency' is never measured, the claimed superiority over 'centralized baselines' is never directly tested, and the 'federated' evaluation decomposes into independent per-client retrievers rather than the cooperative pipeline of Section 3.1. The 'system efficiency' numbers come from a cache-hit simulation (Section 4.3), not from measured inference across the edge-cloud architecture. The GEval scores evaluate privacy of redacted summaries, not the quality of the final generated answer. The abstract's broad claim therefore rests on a text-retrieval artifact, not on the framework being proposed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"HyFedRAG proposes a federated retrieval-augmented generation framework for heterogeneous, privacy-sensitive medical data. The architecture combines edge-side local retrievers over text, SQL, and knowledge-graph sources; privacy-preserving summarization via Presidio, Eraser4RAG, and TenSEAL; server-side LLM fusion of de-identified summaries; and a three-tier cache. The paper reports retrieval results on PMC-Patients for a text-only retriever (Table 1), per-modality retrieval (Table 2), GEval privacy scores for redacted outputs, and a cache simulation that is claimed to reduce inference latency by up to 80%. The central claim is that HyFedRAG outperforms centralized baselines and existing methods in retrieval quality, generation consistency, and system efficiency.","tokens_in":10778,"tokens_out":5773,"duration_ms":57221,"significance":"The architecture addresses a real and timely problem: enabling RAG across heterogeneous, privacy-protected medical data without centralizing raw records. The edge-cloud design, the combination of multiple privacy tools, and the multi-tier cache are sensible components. If supported by a rigorous end-to-end evaluation, the framework could be a useful contribution. As it stands, however, the experiments exercise only individual components—a text retriever, per-modality retrievers, a privacy scorer, and a cache simulation—rather than the proposed federated pipeline. Moreover, the reported retrieval gains rely partly on test-set parameter selection. The paper includes no code or reproducibility package. The demonstrated significance is therefore substantially lower than the claims in the abstract and conclusion.","major_comments":[{"comment":"The central claim that HyFedRAG outperforms centralized baselines in generation consistency and system efficiency is not tested end-to-end. Section 3.1 describes a full pipeline with multiple clients, local anonymized summarization, and server-side fusion, but no experiment instantiates this configuration. Table 1 reports only text retrieval against text retrievers; Table 2 reports per-modality retrieval for Text, SQL, and KG separately. There is no multi-client setup, no server-side fusion step, no generation output, and no comparison to a centralized RAG system over the same heterogeneous data. Consequently the abstract's claims about 'generation consistency' and outperforming centralized baselines are unsupported by the presented evidence.","section":"Section 3.1, Section 4 Experiments, Tables 1-2"},{"comment":"The fusion weight alpha in Eq. (1) is selected from the evaluation set. Section 4.3 states that 'performance peaks at alpha=0.8' on MRR and nDCG@10, and Figure 7 displays the retrieval metrics as a function of alpha. The Table 1 'HyFedRAG(text)' results appear to use this peak value. No validation split or model-selection procedure is described, and the baselines are not given the same test-set tuning. This makes the reported relative gains a test-set-fitted quantity and overstates the generalizable improvement.","section":"Section 4.3, Figure 7, Eq. (1)"},{"comment":"The privacy-utility tradeoff is not measured for the downstream generation task. Section 3.4 asserts that local LLMs produce 'de-identified, semantically rich representations suitable for global reasoning,' but Figure 4 only reports GEval privacy scores of redacted outputs. The paper does not measure whether anonymized summaries preserve enough information for the server-side LLM to answer queries correctly, nor does it compare generation quality with and without anonymization. Table 2 already shows large retrieval degradation for SQL and KG formats, indicating that format transformations can lose semantics; the additional effect of de-identification on answer quality is left unquantified, so the privacy-preservation claim does not establish utility.","section":"Section 3.4, Figure 4, Table 2"},{"comment":"The 80% latency reduction claim is based on a simulation, not on measurements of the deployed HyFedRAG system. Section 4.3 says query sequences were 'generated via a random-walk process' and that a hierarchical cache was 'employed' to record hit/miss rates. Figure 6 reports cache hit rates, and Figure 5 is presented as 'inference latency comparison,' but no actual end-to-end latency measurement over the edge-cloud architecture is reported. The cache simulation parameters (100 warm-up, 500 test requests, LRU policies, prefetch depths) are also not validated against real workloads. The 'system efficiency' contribution is therefore not empirically established.","section":"Section 4.3, Figures 5-6"}],"minor_comments":[{"comment":"The row labeled 'Relative gain' gives +11.87, +0.52, +17.21, but these are absolute percentage-point differences (e.g., 39.63 - 27.76 = 11.87), not relative gains. The text in Section 4.2 also says '1.87 %' where Table 1 shows 11.87. Please correct the labeling and the numbers.","section":"Table 1, Section 4.2"},{"comment":"The text says that 'MRR falls to 0.54 and nDCG@10 to 0.63' when alpha=1.0, but Table 1 reports MRR and nDCG@10 on a percentage scale (39.63% and 41.33%). The units are inconsistent and should be unified.","section":"Section 4.3"},{"comment":"The paper lists Eraser4RAG and TenSEAL as supported privacy tools and counts them as a contribution, but the experiments only evaluate Presidio ('Anonymization is performed using the Presidio privacy protection toolkit'). No results are reported for the other two tools, so their effectiveness is unverified.","section":"Section 3.4, Section 4.2"},{"comment":"Figures 4 and 5 are referenced without numerical values, error bars, or axis definitions. The claimed improvements would be easier to assess if the underlying numbers and evaluation protocol were reported.","section":"Figures 4-6"},{"comment":"The cache simulation uses ad-hoc parameters (100 warm-up and 500 test requests, random-walk restart, dwell, and session-memory mechanisms). No sensitivity analysis or justification is given for these choices, and it is unclear how well they approximate real clinical query workloads.","section":"Section 4.3"}],"recommendation":"reject","confidential_remarks":"The manuscript's main contribution is an architecture, but the evaluation does not test that architecture. The missing end-to-end federated experiment, the lack of generation-quality metrics, and the test-set selection of alpha are fundamental gaps that would require substantial new experiments and analysis to address. A minor revision cannot fix these issues; a new submission with a genuine multi-client evaluation, centralized baselines, and proper parameter selection would be needed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Qian et al. propose HyFedRAG, a federated RAG framework that combines Flower-based orchestration with three retrieval backends (text, SQL, KG), three privacy tools (Presidio, Eraser4RAG, TenSEAL), and a three-tier cache. The architecture is plausible, and the engineering is real—they actually built the stack, and the three-tier cache idea (local features, intermediate representations, and final outputs) is a reasonable way to cut redundant LLM calls. The text-only hybrid retriever, fusing TF-IDF with a BGE dense retriever and FlagReranker, posts large gains over PMC-Patients baselines (MRR 39.63 vs 27.76 for RRF, nDCG@10 41.33 vs 24.12). That is a genuine result for the hybrid scoring scheme.\n\nThe problem is that the abstract and conclusion claim the full federated framework outperforms centralized RAG in retrieval quality, generation consistency, and system efficiency. No experiment runs the full pipeline. Table 1 evaluates only the text module in isolation; Table 2 gives per-modality retrieval for SQL and KG, which drop sharply (MRR 23.01 and 9.79). There is no multi-client setup where clients with different modalities generate anonymized summaries and a server LLM fuses them into an answer. Generation consistency is never measured; the GEval scores in Figure 4 assess privacy of redacted summaries, not final answer quality. The 80% latency reduction comes from a cache simulation over synthetic query sequences, not from real inference. And the alpha in the hybrid retriever is tuned on the test set, as Figure 7 shows, so even the text-retrieval number is a fitted best case. The authors do acknowledge the structured-format degradation in their own analysis, which is honest, but they don't address the gap between component-level evaluation and system-level claims.\n\nThis is not a paper without merit. The architecture and the hybrid retrieval idea are worth discussing, and the text result suggests something real. But the central claim is load-bearing and unsupported. A serious reviewer would need an end-to-end federated experiment with at least two heterogeneous clients, a centralized baseline, error bars or multiple runs, and a metric on the final generated answer. Without that, the paper oversells what it demonstrates.\n\nI would not desk-reject it—the topic is relevant, and the authors could plausibly add the missing experiment—but as submitted, acceptance would be imprudent.","headline":"HyFedRAG builds a plausible federated RAG architecture and shows strong text-only retrieval numbers, but the headline claim about the full framework is not supported by any end-to-end experiment.","tokens_in":11299,"tokens_out":3991,"would_cite":false,"duration_ms":43888,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HyFedRAG proposes that a federated edge-cloud RAG pipeline, where clients keep raw patient data local and send only de-identified summaries, can outperform centralized RAG on heterogeneous, privacy-sensitive medical data while cutting infer","keywords":["federated retrieval-augmented generation","heterogeneous data","privacy-preserving summarization","edge-cloud collaboration","three-tier caching","clinical decision support","PMC-Patients","hybrid retrieval"],"falsifier":"Compare end-to-end retrieval and generation accuracy on PMC-Patients queries when the server receives HyFedRAG's anonymized summaries versus when it receives raw (or minimally redacted) records; if the accuracy gap is large, the federated privacy design cannot deliver the claimed performance, and if the gap is negligible, the assumption holds.","tokens_in":10399,"feed_emoji":"🏥","tokens_out":8508,"duration_ms":82718,"temperature":0.7,"pith_summary":"HyFedRAG claims that a federated, edge-cloud retrieval-augmented generation (RAG) pipeline can handle heterogeneous and privacy-sensitive medical data—SQL tables, knowledge graphs, and free text—without ever moving raw patient records to a central server. The key move is to run retrieval and de-identified summarization locally on each client, then have a central server fuse only those sanitized summaries into a global answer. On the PMC-Patients benchmark, the paper reports that this design outperforms centralized retrieval baselines on MRR, P@10, and nDCG@10, achieves higher privacy scores under an LLM-based evaluator, and cuts end-to-end inference latency by roughly 80% through a three-tier cache. A sympathetic reader would care because current RAG systems assume a single homogeneous repository, while real clinical data is split across institutions, formats, and privacy regulations.","feed_headline":"Federated RAG beats centralized baselines on private medical data","feed_subtitle":"HyFedRAG retrieves across SQL, graphs, and text without sharing raw records—and answers faster.","key_machinery":"The load-bearing mechanism is an edge-cloud collaboration in which each client runs format-specific retrieval (a sparse-dense hybrid with reranking for text, entity matching plus graph traversal for knowledge graphs, and full-text search plus deep reranking for SQL) and then a local LLM produces a privacy-preserving summary of the retrieved records. Three anonymization layers are offered—named-entity masking for coarse de-identification, attribution-aware erasure that removes spans not needed for the query, and homomorphic encryption for feature-level protection—so that only sanitized summaries leave the client. A server-side LLM fuses these summaries into the final answer. The efficiency co","core_discovery":"On its own terms, the paper's central claim is that privacy and performance do not have to trade off in federated RAG: if each participating client converts its raw, format-specific data into standardized, de-identified summaries with a local LLM, a cloud-side LLM can fuse those summaries and produce answers at least as good as centralized systems that see the raw data. The evidence offered is a set of experiments on PMC-Patients: HyFedRAG's text pipeline reaches 39.63 MRR, 7.48 P@10, and 41.33 nDCG@10, against a best baseline of 27.76 MRR, 6.96 P@10, and 24.12 nDCG@10; the SQL and knowledge-graph variants degrade substantially (23.01 and 9.79 MRR respectively), which the paper attributes to","pith_inferences":["The strongest unmeasured quantity is information loss: the paper does not directly compare raw records against their de-identified summaries for downstream answer quality, so a natural next experiment is to measure retrieval and generation accuracy as a function of summary compression.","The three-tier caching design is not specific to medicine; it should transfer to any federated RAG deployment, and a benchmark on legal or financial heterogeneous data would test whether the 80% latency reduction generalizes.","The privacy evaluation relies on an LLM judge scoring redaction output; a stricter test would attempt re-identification attacks against the summaries or measure residual private information in the fused answer itself.","Because structured formats perform worst, a plausible extension is format-aware summary generation—learning what to preserve from SQL schemas or graph neighborhoods rather than applying the same text-oriented anonymization pipeline to all modalities."],"forward_implications":["Cross-institution clinical question answering can proceed without moving raw records: only de-identified summaries cross the network, which is a precondition for GDPR/HIPAA-style settings.","Hybrid retrieval with a tunable fusion weight improves over pure lexical or pure semantic retrieval; the paper finds the optimum near alpha=0.8.","Structured data formats (SQL, knowledge graphs) lose retrieval quality compared with free text, so federated RAG over structured sources needs additional semantic augmentation to be clinically useful.","Three-tier caching with neighborhood prefetching can reduce latency by roughly 80% in simulated federated workloads, making edge-cloud RAG practical on modest hardware.","The same architecture can be assembled from off-the-shelf components: local embedding models, rerankers, local LLMs, and a federated orchestration layer."],"supporting_citations":[{"why":"Supplies the PMC-Patients dataset, the query task, and the leaderboard baselines (RRF, BM25, DPR variants) that HyFedRAG is measured against.","marker":"Zhao et al. 2022"},{"why":"Supplies Flower, the federated learning platform used to implement the edge-cloud client-server workflow.","marker":"Beutel et al. 2020"},{"why":"Supplies the bge-reranker-v2-m3 reranking model used in the text and knowledge-graph retrieval pipelines.","marker":"Li et al. 2023"},{"why":"Supplies Llama-3.1-8B-Instruct, the local LLM used for privacy-preserving summarization and for knowledge-graph triple extraction.","marker":"Dubey et al. 2024"},{"why":"Supplies the Presidio PII masking toolkit used as the main anonymization step before summaries leave the client.","marker":"Microsoft 2021"},{"why":"Supplies Eraser4RAG, the attribution-aware erasure method used to remove query-irrelevant sensitive spans.","marker":"Wang et al. 2025b"},{"why":"Supplies TenSEAL, the homomorphic encryption library used to compute and transmit encrypted embeddings.","marker":"Benaissa et al. 2021"},{"why":"Supplies the G-Eval scoring method (via DeepEval) used to judge generation consistency and privacy scores.","marker":"Liu et al. 2023"},{"why":"Supplies Gemma-2-9B-IT, the second local LLM used to test whether the framework's results depend on the summarization model.","marker":"Team et al. 2024"}],"fun_headline_variants":["Federated RAG outperforms centralized on private medical data","HyFedRAG: Privacy-first RAG that beats cloud baselines","Edge-cloud RAG handles heterogeneous health data without leaks","HyFedRAG: Federated RAG for private, mixed healthcare records"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that locally produced de-identified summaries preserve enough clinical meaning for the server-side LLM to fuse them into correct global answers; the paper never directly measures the information lost between raw records and those summaries.","fun_headline_variants_meta":{"raw":{"variants":["Federated RAG outperforms centralized on private medical data","HyFedRAG: Privacy-first RAG that beats cloud baselines","Edge-cloud RAG handles heterogeneous health data without leaks","HyFedRAG: Federated RAG for private, mixed healthcare records"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000389,"raw_usage":{"total_tokens":1947,"prompt_tokens":867,"completion_tokens":1080,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":1006}},"tokens_in":611,"tokens_out":1080,"duration_ms":12020,"temperature":1.0,"reasoning_tokens":1006,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T23:36:01.795482+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare end-to-end retrieval and generation accuracy on PMC-Patients queries when the server receives HyFedRAG's anonymized summaries versus when it receives raw (or minimally redacted) records; if the accuracy gap is large, the federated privacy design cannot deliver the claimed performance, and if the gap is negligible, the assumption holds.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Presidio PII masking toolkit used as the main anonymization step before summaries leave the client."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the G-Eval scoring method (via DeepEval) used to judge generation consistency and privacy scores."}],"review_version":1}