{"id":"036e42ae-7e51-4e15-ae80-c3b407dee03c","arxiv_id":"2506.09200","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"A new library, FedRAG, provides centralized and federated fine-tuning for RAG systems, with a lightweight experiment showing a RALT-based accuracy gain on MMLU global facts.","lead":"FedRAG is a new open-source Python framework that fine-tunes retrieval-augmented generation (RAG) systems in centralized or federated setups. It wraps existing methods like RALT and LSR with a simple API and connections to HuggingFace, Qdrant, and LlamaIndex.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The federated training path -- the framework's novel claim -- is never executed or validated; 'seamless conversion' rests on an unverified flwr backend, not on evidence.","rationale":"You cannot claim a framework works for 'federated architectures' without executing the federated path. The strongest claim, that no other tool offers simple centralized-to-federated conversion for RAG fine-tuning, is a usability claim; usability of the conversion cannot be assessed from a code sketch. The reader correctly identified this as the weakest assumption. I agree and would keep the CONDITIONAL verdict: the concern is real but addressable by adding a federated experiment or at least a unit test of the flwr integration. The released code and containerized knowledge store are meaningful support, and the paper does not overclaim the learning results (it calls Appendix A lightweight), but the abstract's 'supports state-of-the-art fine-tuning methods' is partially undercut by the roadmap listing ReSearch as future work. This is secondary; the primary issue remains the unvalidated federated backend.","tokens_in":9179,"tokens_out":3228,"duration_ms":33511,"concrete_test":"Run the Figure 3 federated path end-to-end with two clients on a small fixed dataset (e.g., a 200-example subset of Web Questions), using the released repository and a fixed seed. Verify that (1) both flwr clients and the server complete without error, (2) the returned aggregated model's weights differ from initialization and equal the FedAvg of the two client updates, and (3) the federated model's MMLU global-facts exact match is within the run-to-run sampling noise of the centralized RALT result in Table 3. If such a run is infeasible, inspect the fl_task server and client implementation and unit-test FedAvg parameter aggregation on a toy model with two diverging client updates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central value proposition (Sections 2.2 and 3.2) is that FedRAG makes federated RAG fine-tuning simple through a 'seamless conversion from centralized to federated training tasks.' The only support for this is the Figure 3 code sketch: manager.get_federated_task() returns an fl_task whose server and clients are launched with flwr.start_server and flwr.start_client. No federated experiment, log, or ablation appears anywhere in the paper or appendix; the federated path is never run. This matters because federated averaging over HuggingFace trainers is nontrivial: client and server must agree on weight initialization, optimizer state (e.g., Adam moments, QLoRA adapters), and freezing logic for the non-trained retriever or generator, and flwr's default FedAvg must be compatible with the manager's per-client setup. A bug or mismatch in any of these would break the central claim even though the centralized trainer works. The only empirical support, Appendix A, is two noisy runs on 100 MMLU global-facts items with baseline exact match varying 17-27 and RALT runs 27-34, with no seeds or error bars; this is too thin to establish that the framework 'successfully executes' even centralized RAG fine-tuning beyond wrapping HuggingFace's Trainer. Because the federated path is the differentiator, its complete lack of validation is the load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces FedRAG, a library framework for fine-tuning retrieval-augmented generation (RAG) systems in both centralized and federated settings. It describes a modular design (core, evals, FL tasks, generators, knowledge stores, retrievers, trainers), shows usage patterns via code snippets, and reports a lightweight RA-DIT-inspired experiment in Appendix A in which a Llama-2-7B generator is fine-tuned with the RALT objective using QLoRA and evaluated on the MMLU global-facts subset. The paper claims that FedRAG offers state-of-the-art fine-tuning methods, seamless conversion from centralized to federated training, and fills a critical gap in RAG tooling.","tokens_in":9404,"tokens_out":5926,"duration_ms":55710,"significance":"If the framework delivers what is claimed, it would be a valuable systems contribution: a single API for retriever/generator fine-tuning, integrated with HuggingFace, Unsloth, Qdrant, and LlamaIndex, plus a straightforward route to federated training. The release of library code, a containerized knowledge store, and experimental code are concrete strengths that aid reproducibility. However, the current evidence is far too thin to support the central claims. The federated training path is never executed; the only experiment is two noisy runs of a single fine-tuning method on 100 examples; and the 'critical gap' claim is not supported by a systematic survey of existing tools. The paper is therefore best viewed as a framework proposal with an initial proof-of-concept, rather than a validated system.","major_comments":[{"comment":"The claim of a 'seamless conversion' from centralized to federated RAG fine-tuning is the paper's distinguishing feature, but the federated path is never exercised. Figure 3 only shows that manager.get_federated_task() returns an object with server and client launched via flwr.start_server and flwr.start_client; no experiment, log, or ablation with this path appears in the paper or appendices. Federated averaging over HuggingFace trainers with QLoRA adapters, frozen retriever/generator, and optimizer state is non-trivial, and a mismatch in initialization, adapter handling, or freezing logic would silently break the central claim. I recommend adding a minimal federated experiment with at least two clients on the same 100-item MMLU subset, reporting accuracy and comparing with the centralized run.","section":"Section 4.2, Figure 3"},{"comment":"The empirical validation consists of two runs on 100 MMLU global-facts examples, with baseline exact match of 17.0 and 27.0 and RALT exact match of 27.0 and 34.0. The between-run spread (10 and 7 points) is as large as the reported mean improvement (8.5 points). The paper reports no random seeds, hyperparameters (learning rate, epochs, batch size, LoRA rank/alpha), or evaluation settings, despite attributing variability to 'sampling parameters used for generation' (Section A.3). This is insufficient to substantiate Section 5's conclusion that the framework 'can be used to successfully and flexibly execute RAG fine-tuning tasks.'","section":"Appendix A, Table 3"},{"comment":"The abstract and introduction claim that FedRAG fills 'a critical gap in available tools' and supports 'state-of-the-art fine-tuning methods,' but the evidence does not match these claims. Only RALT generator fine-tuning is demonstrated; LSR, the combined RA-DIT sequence, and retriever fine-tuning are never run, and ReSearch is only listed on the roadmap (Table 4). Additionally, the related work (Section 2.2) is too narrow to establish that no existing framework offers similar functionality; a comparison with general federated LLM fine-tuning platforms (e.g., FedML, NVIDIA FLARE, OpenFL) is missing. Please either temper the claims or provide evidence for them.","section":"Sections 1, 2.2, and 5"}],"minor_comments":[{"comment":"The import statement in Figure 2 is missing commas between the trainer class names, reading 'HuggingFaceTrainerForRALT HuggingFaceTrainerForLSR' instead of a proper Python import list.","section":"Section 4.2, Figure 2"},{"comment":"The MMLU evaluation with exact match is ambiguous; please clarify whether the model outputs a multiple-choice letter or a free-form answer, and how exact match is computed.","section":"Section A.2"},{"comment":"The few-shot examples are said to be 'randomly drawn from the validation split and held fixed,' but no seed or the actual examples are given, which hinders reproducibility.","section":"Section A.2"},{"comment":"The sentence 'Of the integrations listed in Table 2, only the LlamaIndex integration had not been represented in the preceding patterns' is grammatically unclear; it likely means that the LlamaIndex bridge is not shown in Figures 1-4.","section":"Section 4.3"},{"comment":"The reference to the library code appears only in a footnote; consider stating the repository URL and a versioned release in the main text for reproducibility.","section":"Footnote 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is more of a library announcement than a research paper. The central differentiator (federated fine-tuning) is entirely unvalidated, and the centralized experiment is too noisy to be conclusive. I would be willing to reconsider after the authors add a federated experiment and run the centralized evaluation with proper controls and error bars. There is also a self-citation concern: reference [11] includes the first author and is cited in support of the related-work gap, which is acceptable but should be validated independently."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: FedRAG is a genuine systems contribution — a clean API for fine-tuning RAG retrievers and generators, with HuggingFace/Unsloth/LlamaIndex/Qdrant integrations and a claimed one-line conversion to federated training. If you do applied RAG work, this is worth knowing about. What is actually new is the integration and the federated task interface; the underlying methods (RALT, LSR, RA-DIT) are prior work, and the paper says so honestly.\n\nThe strongest part is the design: the abstractions (RAGSystem, trainer manager, fl_task extraction) look sensible, and the modular layout is easy to navigate. The appendix shows a real RALT run on MMLU global facts, with code and a containerized knowledge store released. That is reproducible evidence that the centralized path works, at least for a small example. The self-citation to LlamaIndex Networks is incidental and not load-bearing.\n\nThe soft spots are exactly where the stress-test lands. The federated path — the paper's main differentiator, advertised as \"seamless conversion\" — is never executed. Figure 3 is just a code sketch; there is no federated experiment, no log, no ablation. Federated averaging over HuggingFace trainers is nontrivial (optimizer state, QLoRA adapters, freezing logic), so an unverified flwr backend cannot carry the claim. The centralized evidence is also thin: two runs on 100 items, baselines 17 and 27, RALT runs 27 and 34, no seeds or error bars. That is enough to show the pipeline runs, not enough to establish \"significant performance gains\" — the authors hand-wave the variability as sampling noise. Finally, \"state-of-the-art methods\" overstates what is shipped: ReSearch is on the roadmap, and LSR gets no experiment here.\n\nNone of this is disqualifying for a systems paper. It is a framework, not a new learning result, and the code is out there. A serious referee should engage: the right revision adds either a small federated run (even with synthetic clients) or a clear statement that the federated path is untested, plus more seeds and hyperparameters. If you work with RAG fine-tuning, this is a plausible tool to try; the paper's science is thin but the engineering looks real.\n\nRecommendation: send to peer review with a request for a federated validation experiment.","headline":"A clean, well-integrated RAG fine-tuning framework with released code, but the federated path — the headline feature — is never executed and the centralized validation is two noisy runs on 100 examples.","tokens_in":9989,"tokens_out":2105,"would_cite":true,"duration_ms":20634,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FedRAG claims to be a framework that makes fine-tuning RAG retrievers and generators straightforward in both centralized and federated settings, with seamless conversion between them and deep integration with the HuggingFace-based tooling…","keywords":["retrieval-augmented generation","fine-tuning","federated learning","RAG framework","RA-DIT","RALT","LSR","LLM"],"falsifier":"Run the framework's federated path with two or more clients on disjoint subsets of Web Questions and compare exact-match accuracy on the MMLU global facts split against the centralized RALT result; if the federated model fails to train or clearly underperforms the 30.5 average, the claim of seamless federated conversion is falsified.","tokens_in":8960,"feed_emoji":"🧩","tokens_out":5631,"duration_ms":55419,"temperature":0.7,"pith_summary":"The paper introduces FedRAG, a software framework whose central claim is that fine-tuning retrieval-augmented generation systems—both the retriever and the generator—can be done through one clean set of abstractions, and that the same centralized training task can be converted into a federated one with minimal code changes. If this works as claimed, practitioners would be able to adapt RAG systems to domain-specific data, including data that cannot be centrally collected, without learning a new training stack. The framework wraps state-of-the-art objectives like RALT and LSR, integrates with popular tools, and exposes a trainer manager that orchestrates which model is updated while the other is frozen. The paper's own validation is a lightweight centralized experiment: QLoRA fine-tuning with the RALT objective improved average exact match on the MMLU global facts subset from 22.0 to 30.5, while the federated path is described but never executed.","feed_headline":"FedRAG makes RAG fine-tuning drop-in for central and federated setups","feed_subtitle":"Library wraps fine-tuning trainers and converts centralized RAG tasks to federated learning with minimal code changes.","key_machinery":"The central object is the RAGSystem, composed of a KnowledgeStore, a Retriever, and a Generator, along with a TrainerManager that orchestrates fine-tuning: it prepares the target model for training, freezes the other model, and exposes a train() method and a get_federated_task() method. The fine-tuning methods implemented are RALT (Retrieval-Augmented Language Model Training), which teaches the generator to use retrieved context, and LSR (Language Model Supervised Retriever Training), which minimizes the KL divergence between retriever scores over retrieved chunks and generator target probabilities. The manager's ability to emit a federated task object is the mechanism that carries the paper's central claim of seamless centralized-to-federated conversion.","core_discovery":"The paper's central claim is that FedRAG provides a single workflow spanning the whole RAG fine-tuning lifecycle: assemble a RAGSystem from a retriever, a generator, and a knowledge store; fine-tune either component with objectives such as RALT (generator training on retrieved context) or LSR (retriever training via distribution matching); and then convert the centralized trainer into a federated task by extracting an FL task object that plugs into a Flower-based server and client. The authors argue this fills a gap because existing RAG frameworks focus on assembly and inference while leaving fine-tuning and federated conversion to ad hoc user effort.","pith_inferences":["The paper does not show an executed federated experiment, so the strongest practical payoff—training on private or fragmented data—rests on an untested assumption; a two-client run on disjoint data would be the natural next check.","The reported MMLU gain comes from 100 test items and two noisy runs, so the 22-to-30.5 improvement is suggestive but not a stable estimate; a multi-seed run on a larger benchmark would settle the effect size.","The same trainer-manager pattern could serve as a testbed for comparing generator-side objectives against retriever-side losses within identical RAG configurations, isolating which component actually drives downstream gains.","The roadmap's planned ReSearch-style reinforcement-learning trainer would extend the framework from supervised fine-tuning to reasoning-with-search, a larger design space the current centralized experiment does not touch."],"forward_implications":["A user can assemble a RAG system, fine-tune the retriever or generator with RALT or LSR, and then extract an FL task to run the same training federated across clients with minimal code changes.","The framework integrates with HuggingFace models and datasets, Unsloth for fast generator fine-tuning, Qdrant for the knowledge store, and LlamaIndex for inference bridging.","The lightweight RALT reproduction improved average exact match on the MMLU global facts subset from 22.0 to 30.5, consistent with the direction of earlier RA-DIT results.","Because the components are modular, new trainers, losses, benchmarks, and knowledge stores can be added without rewriting the core pipeline.","If the federated conversion works, FedRAG would give organizations a practical route to fine-tuning RAG systems on data that must remain decentralized."],"supporting_citations":[{"why":"Defines RALT and LSR and the RA-DIT sequential fine-tuning recipe that the paper's experiment is designed to reproduce.","marker":"[22]"},{"why":"Supplies federated averaging, the algorithm that the framework's federated task conversion relies on to combine client updates.","marker":"[25]"},{"why":"Provides the DRAGON+ retriever used in the centralized fine-tuning and evaluation pipeline.","marker":"[21]"},{"why":"Supplies the Llama2-7B model used as the quantized generator in the paper's experiment.","marker":"[32]"},{"why":"Provides QLoRA, the parameter-efficient fine-tuning method applied to the generator during RALT training.","marker":"[9]"},{"why":"Provides the MMLU benchmark, specifically the global facts subset, used to measure exact-match performance.","marker":"[15]"},{"why":"Supplies the Wikipedia passage dump used as the source of the 10M chunks loaded into the knowledge store.","marker":"[17]"},{"why":"Supplies the Web Questions dataset used as the fine-tuning training data.","marker":"[2]"},{"why":"Describes RAFT, a comparable generator fine-tuning approach cited as part of the state-of-the-art methods the framework supports.","marker":"[36]"}],"fun_headline_variants":["FedRAG: fine-tune RAG systems centrally or federated with minimal code","FedRAG: unified RAG fine-tuning for central and federated architectures","FedRAG bridges the gap in RAG fine-tuning from central to federated","FedRAG: one framework for RAG fine-tuning in any deployment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The seamless conversion from centralized to federated fine-tuning works as advertised, but the paper never executes a federated run, so the entire federated benefit rests on that untested assumption.","fun_headline_variants_meta":{"raw":{"variants":["FedRAG: fine-tune RAG systems centrally or federated with minimal code","FedRAG: unified RAG fine-tuning for central and federated architectures","FedRAG bridges the gap in RAG fine-tuning from central to federated","FedRAG: one framework for RAG fine-tuning in any deployment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000866,"raw_usage":{"total_tokens":3674,"prompt_tokens":783,"completion_tokens":2891,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":399,"completion_tokens_details":{"reasoning_tokens":2806}},"tokens_in":399,"tokens_out":2891,"duration_ms":20119,"temperature":1.0,"reasoning_tokens":2806,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:53:56.275322+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the framework's federated path with two or more clients on disjoint subsets of Web Questions and compare exact-match accuracy on the MMLU global facts split against the centralized RALT result; if the federated model fails to train or clearly underperforms the 30.5 average, the claim of seamless federated conversion is falsified.","supporting_citations":[{"cited_title":"Ra- dit: Retrieval-augmented dual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Defines RALT and LSR and the RA-DIT sequential fine-tuning recipe that the paper's experiment is designed to reproduce."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies federated averaging, the algorithm that the framework's federated task conversion relies on to combine client updates."},{"cited_title":"Semantic parsing on freebase from question-answer pairs","cited_arxiv_id":null,"evidence_quote":"Supplies the Web Questions dataset used as the fine-tuning training data."},{"cited_title":"Raft: Adapting language model to domain specific rag","cited_arxiv_id":null,"evidence_quote":"Describes RAFT, a comparable generator fine-tuning approach cited as part of the state-of-the-art methods the framework supports."}],"review_version":1}