{"id":"2b7d5f94-7df0-4f17-9efc-fc2ac1e17a46","arxiv_id":"2602.00911","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"Structured compendium artifacts can serve as the federated exchange unit for tool-routing LLMs, preserving accuracy across clients and model families without sharing weights or raw data.","lead":"SYNAPSE proposes exchanging structured \"compendium\" documents—schema-validated descriptions of tool behavior—between federated LLM clients, rather than sharing weights, prompts, or raw data. The paper reports strong tool-routing accuracy on math benchmarks with far lower communication than weight-sharing, plus formal-looking privacy and convergence guarantees.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline cross-family transfer claim is absent from the body: Appendix D only reports LLaMA 3.2-11B→3.2-3B prompt-transfer deltas (+0.03/+0.15/−0.08); no Mistral 7B, GPT-4o, or four-family compendium experiment appears.","rationale":"The reader's weakest assumption is the contraction premise in Theorem A.2. That is a genuine weakness, but it is secondary: the theorem is already phrased conditionally, and Figure 7 provides empirical convergence evidence under fixed noise. The more load-bearing problem is that the abstract's signature claim — cross-architecture transfer of a single compendium across four LLM families with ~2 pt loss — cannot be found in the body. Appendix D's transfer table involves only LLaMA 3.2-11B to LLaMA 3.2-3B and reports mixed deltas, including an 8-point drop on GSM8k. This missing evidence directly undermines the claimed superiority over weight-sharing federation. I keep the verdict at CONDITIONAL rather than REJECT because the framework is concretely implemented, baseline comparisons and hyperparameters are reported, and the missing transfer experiment is a well-defined check that could be added or pointed to. If the audit and reproducibility run fail to produce four-family transfer with ~2 pt loss, the correct disposition would be reject; if they succeed, the central concern is resolved. The odd inserted line in Appendix A.4 and the abstract/body inconsistency reinforce the need for a careful provenance check, but the scientific issue is the missing transfer experiment.","tokens_in":20247,"tokens_out":10191,"duration_ms":103544,"concrete_test":"Audit every table and figure in Section 4 and Appendices D-F for a cell/panel reporting accuracy from a Mistral 7B or GPT-4o client after transfer of the same global compendium. If no such cell exists, the four-family transfer claim is unsupported by the manuscript. To test whether the claim is true rather than merely absent, use the supplied anonymized repository to rerun Appendix D's protocol with Mistral 7B and GPT-4o as target LLMs on the GSM8k/BBH federated splits, measuring global accuracy after transferring the LLaMA-trained compendium; if the average drop across families is not approximately 2 pt (and specifically if Table 4's -0.08 on GSM8k is representative), the abstract's claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central empirical claim, as stated in the supplied abstract, is that a single compendium transfers across four LLM families (LLaMA 3.1-8B, LLaMA 3.2-3B, Mistral 7B, GPT-4o) with approximately 2 pt loss. This result is the basis for claiming an advantage over weight-sharing federation, which cannot transfer across architectures. Yet the manuscript body contains no such experiment. The only transfer-related section, Appendix D, reports optimized-prompt transfer from LLaMA 3.2-11B to LLaMA 3.2-3B on three tasks, with performance changes +0.03, +0.15, and -0.08 (Table 4). That is not four families, is not a compendium-transfer measurement, and does not show an approximately 2 pt loss. No Mistral 7B or GPT-4o transfer result appears anywhere in the body. The formal results in Appendix A are also conditional (e.g., Theorem A.2 assumes the routing operator is a contraction with L<1, and the abstract concedes this fails for one distribution; the DP proof for text masking is a sketch), but those caveats are explicitly stated. The gap between the abstract's cross-family claim and the reported experiments is not a caveat; it is missing evidence for the paper's headline capability. Because the abstract and body disagree on the core result, the strongest claim is currently unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SYNAPSE, a federated knowledge-exchange framework in which clients with frozen, heterogeneous LLMs share structured JSON 'compendiums' containing tool metadata, usage scenarios, precautions, prompt templates, and structured annexes, rather than model weights or raw prompts. Compendiums are aggregated hierarchically (client-edge-server), optionally refined with TextGrad prompt optimization, and used at inference for embedding retrieval plus LLM reranking to route queries to tools. The manuscript claims a typed merge operator, per-field differential privacy via Laplace noise and adaptive masking, routing-stability/convergence guarantees, communication superiority over weight sharing, and cross-LLM-family transfer. The body reports experiments on GSM8k and two BBH tasks, privacy/robustness analyses, and appendices containing theoretical claims and additional experiments.","tokens_in":20744,"tokens_out":9773,"duration_ms":108875,"significance":"If fully supported, the idea of typed compendium artifacts with per-field privacy, schema-aware merging, and cross-architecture transfer would be a useful contribution to federated LLM tool routing. The communication-cost comparison to fp32 weight sharing is in-principle favorable, and the paper provides an anonymous code URL and detailed appendices that aid reproducibility. However, the central cross-family transfer claim in the abstract is not present in the body, and the formal privacy and convergence theorems are either conditional on unproved assumptions or incomplete. The significance of the claimed results cannot be assessed until those gaps are addressed.","major_comments":[{"comment":"The abstract claims: 'A single compendium transfers across four LLM families (LLaMA 3.1-8B, LLaMA 3.2-3B, Mistral 7B, GPT-4o) with approximately 2 pt loss.' No experiment in the body or appendices reports such a transfer. Appendix D, Table 4 reports only prompt transfer from LLaMA 3.2-11B to LLaMA 3.2-3B on three tasks, with changes +0.03, +0.15, and -0.08; no Mistral 7B or GPT-4o compendium-transfer result appears, and no 2-pt-loss figure is given. Table 3 is a centralized-vs-federated comparison, not a cross-family transfer. Since the cross-architecture transfer claim is the stated advantage over weight-sharing federation, this is missing evidence for the paper's headline capability. The manuscript must either add the four-family experiment or substantially revise the abstract and conclusions to match the evidence.","section":"Abstract vs. Appendix D"},{"comment":"Theorem A.2 assumes the routing operator R is a Lipschitz contraction with L<1 on a complete metric space of routing decisions. This is exactly the routing-stability property to be established; Banach's theorem then yields the conclusion without additional content. The manuscript does not measure, bound, or prove the contraction constant for the actual SYNAPSE routing operator, and the abstract itself concedes the premise fails for one of the five distributions. The stochastic component is dismissed as a martingale difference sequence with bounded variance, again without verification. As stated, the result is conditional and does not provide the claimed convergence guarantee. A valid argument needs either an explicit contraction estimate or a different mechanism (e.g., diminishing update magnitudes) with assumptions checked on the evaluated distributions.","section":"Appendix A.2, Theorem A.2"},{"comment":"The DP claim for artifact sharing is incomplete. The Laplace mechanism for numeric metadata is standard, but Theorem A.3 asserts that adaptive text masking 'modeled as randomized response or local differential privacy on tokens' satisfies epsilon-DP. The masking rule in Appendix A, p(w)=min(1, lambda*kappa(w)), depends on a per-token saliency score; no output alphabet or mechanism is specified, and the dependence of p on the token itself means the mask pattern can leak information about that token. Deterministic summarization/truncation is also not DP. The advanced composition formula is standard but cannot be applied until a per-round epsilon for the masking step is established. The theorem should either define a concrete mechanism with a proof, or be replaced by a claim limited to numeric metadata.","section":"Appendix A.3, Theorem A.3"},{"comment":"Theorem A.1's proof assumes the privacy transformation is an epsilon-DP randomized mapping and that epsilon-DP implies a bound on expected text distance delta_s(epsilon). Neither follows from differential privacy definitions; DP constrains output probabilities, not metric distance to the input. The proof also assumes the embedding is Lipschitz and that similarity deviations are Lipschitz in the embedding, with no constants estimated. Consequently the 'bounded embedding distortion' and 'conditional retrieval distortion' claims in the abstract are not established. If the intent is only an empirical distortion characterization, Figure 12 should be presented with error bars and a clear statement that no formal bound is proved.","section":"Appendix A.1, Theorem A.1"},{"comment":"The empirical evaluation is reported as point estimates without repeated-run uncertainty. Table 1 reports client-level spread and std. dev., but these measure client heterogeneity, not run-to-run variability. Tables 3 and 4 report single accuracies or deltas with no confidence intervals or significance tests. Figure 7's convergence trajectories are single examples with fixed/unfixed noise. Without repeated-seed experiments, claims such as 'performance comparable to centralized retrieval-only' and the transfer deltas in Table 4 cannot be assessed. Please provide repeated-run error bars or at least state the number of runs and the variance across runs.","section":"Tables 1, 3, 4 and Fig. 7"}],"minor_comments":[{"comment":"The Weight-Share row is internally inconsistent: the text says 64 GB per client per round, but the table's Avg. Bytes/Client/Round entries (564*10^9 and 864*10^9) do not match the totals or the text. Please reconcile the units and values.","section":"Appendix E, Table 5"},{"comment":"The text contains an unexplained line 'Joy Maa Baglamukhi' after Figure 12, and an incomplete citation 'sum - uid (Chen et al., 2025)'. These appear to be editing artifacts and should be removed or clarified.","section":"Appendix A.4"},{"comment":"The acronym ALT is introduced as 'Adaptive Laplace Text Noise' in Appendix A.4, but the earlier text consistently calls the mechanism 'adaptive text masking'. Please standardize the terminology.","section":"Section 4.2 and Appendix A.4"},{"comment":"Several references have formatting issues, e.g., 'DworkCynthia & RothAaron' lacks a space, and the entry for 'sum-uid' is incomplete. Please run a reference-checking pass.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main risk is not stylistic: the headline four-family transfer claim is absent from the body, and the two formal theorems are not proofs as written. I recommend major revision only because these gaps are potentially addressable by adding experiments and repairing the claims. If the authors cannot supply the missing cross-family experiment or cannot provide a correct DP proof for text masking, the paper should be rejected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the abstract's central empirical claim is not in the paper. The abstract says a single compendium transfers across four LLM families (LLaMA 3.1-8B, 3.2-3B, Mistral 7B, GPT-4o) with ~2pt loss. I read the body looking for that experiment. Appendix D has a centralized-vs-federated comparison on LiveBench using GPT-4o, and a prompt-transfer table from LLaMA 3.2-11B to 3.2-3B. Neither is a compendium transfer across four families, and Mistral never appears. So the headline capability is currently unverified.\n\nWhat is genuinely new: the typed compendium artifact with field-wise merge, per-field privacy, and retrieval-based routing. That's a concrete representation unit beyond raw prompts/examples, and the communication numbers in Appendix E show a real four-orders-of-magnitude reduction vs weight sharing. The framework is described in enough detail to rebuild: datasets, splits, hyperparameters, prompt templates are all there.\n\nThe soft spots are real but some are more important than others. The missing cross-family experiment is the load-bearing one; without it the abstract overclaims. Theorem A.2 is essentially circular: it assumes the routing operator is a contraction and then invokes Banach to conclude convergence. That is not a proof of convergence, and the paper itself concedes the premise fails on one distribution. Theorem A.3's DP guarantee for text masking is asserted rather than proved; saliency-based masking is not a standard DP mechanism and needs careful calibration. The empirical tables have no error bars, and ground-truth tool labels come from dataset provenance, not verified annotation—on these tasks that is almost a trivial heuristic (numeric word problem → mathqa). Those are addressable but they matter.\n\nThe paper is not a waste of time. The compendium idea is worth discussing, and the authors are honest about limitations (honest-but-curious, no Byzantine). But as written, the strongest claim is unsupported.\n\nI'd send this to peer review rather than desk-reject: the mechanism is plausible and the missing experiment is concrete and fixable. A referee should demand the four-family transfer result (or removal of the claim), a real contraction measurement or a more honest stability statement, and a proper DP proof for masking. If those land, this could be a useful methods paper for people building cross-architecture federated tool routing.","headline":"The abstract's headline cross-family transfer claim is not in the body; the typed compendium idea is worth a look, but the paper needs major revision before its central claims are supported.","tokens_in":21145,"tokens_out":2796,"would_cite":false,"duration_ms":29746,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Federated tool routing can be built on typed, schema-validated compendiums rather than on weights or prompts, and a single compendium transfers across LLM families with about two points of accuracy loss.","keywords":["federated learning","tool routing","retrieval-augmented generation","typed artifacts","compendium schema","differential privacy","cross-architecture transfer","communication efficiency"],"falsifier":"Measure the routing operator's round-to-round Lipschitz constant on real client-update trajectories; if it reaches 1 on any distribution, the Banach fixed-point argument in Theorem A.2 cannot deliver almost-sure convergence. Alternatively, run the cross-family transfer experiment with an LLM family held out during federation; if the accuracy drop consistently exceeds the reported ~2 points, the portability claim is falsified.","tokens_in":20183,"feed_emoji":"🔧","tokens_out":5345,"duration_ms":58738,"temperature":0.7,"pith_summary":"This paper claims that the choice of what gets exchanged in federated learning determines what guarantees are even expressible. It proposes typed federated artifacts—schema-validated documents called compendiums that package tool metadata, usage scenarios, precautions, prompt templates, and structured annexes—so that privacy, conflict resolution, and cross-model transfer become well-defined operations instead of heuristics. Concretely, SYNAPSE is a three-tier federation of clients, edge aggregators, and a central server that merges local compendiums into a global tool-routing snapshot, then uses retrieval, LLM reranking, and planning to select tools at inference. The paper reports that one global compendium transfers across four LLM families with about two percentage points of accuracy loss, that communication drops by orders of magnitude versus weight sharing, and that per-field differential privacy on numeric metadata and tunable masking on text have separate, quantified effects on retrieval. A sympathetic reader would care because it offers a path to collaborative tool use among frozen, heterogeneous models without sharing raw data, gradients, or weights.","feed_headline":"Typed tool compendiums let LLMs federate without sharing weights","feed_subtitle":"A structured artifact carries tool knowledge across model families with about two points of accuracy loss.","key_machinery":"The central object is the compendium, a schema-validated JSON artifact whose fields—tool metadata, usage scenarios, precautions, prompt templates, and structured annex—are the units of federation. The typed merge operator performs field-wise conflict resolution during edge and central aggregation; per-field privacy operations, Laplace noise on numeric metadata and adaptive token masking on usage scenarios, are applied before sharing. Routing itself runs as a retrieve-then-rerank-then-plan pipeline, and the formal stability argument treats the routing operator as a contraction on a metric space of routing decisions, invoking Banach's fixed-point theorem to claim almost-sure convergence of too","core_discovery":"The paper's central claim is a change in the unit of exchange: instead of sharing model weights, prompts, or raw examples, clients share compendiums—typed JSON documents holding tool descriptions, usage scenarios, precautions, prompt templates, and a structured annex. Because the schema is explicit, operations like merging, privacy perturbation, and cross-model transfer can be defined field by field. A typed merge operator resolves conflicts per field, Laplace noise gives a formal differential-privacy guarantee on numeric metadata, and retrieval distortion and routing stability are characterized across five distributions, including one where the contraction premise used in the stability proo","pith_inferences":["Beyond the paper: if compendium transfer really holds across LLM families, the artifact could become a portable interchange format for tool knowledge, letting an organization build a routing compendium once and deploy it on whatever model it later adopts.","Beyond the paper: the error analysis suggests retrieval quality is the main bottleneck; a direct extension would test whether replacing the embedding model alone recovers most of the accuracy lost under non-IID splits, independent of the federation protocol.","Beyond the paper: the convergence result is shown for stationary client updates; under non-stationary or adversarial updates the paper's own experiments show oscillation, so a practical deployment would need drift detection—a testable extension rather than a claim of the paper.","Beyond the paper: the DP guarantee covers numeric metadata, while text masking is heuristic; a sharper extension would compute an end-to-end privacy loss across the full pipeline, including the summarization step, and measure it against actual extraction attacks."],"forward_implications":["Federated tool routing no longer requires clients to run the same LLM; frozen heterogeneous models can share a global routing artifact.","Per-field privacy becomes operationally meaningful: numeric metadata receives formal differential privacy while textual scenarios get tunable masking, with masking affecting retrieval more than metadata noise.","Communication during federation drops to kilobytes per client per round instead of gigabytes, enabling bandwidth-constrained deployments.","Routing decisions stabilize under stationary conditions with fixed privacy noise, suggesting a federation can converge on tool selection without sharing gradients."],"fun_headline_variants":["Typed artifacts let LLMs share tools without sharing weights","Federated tool routing via typed compendiums, no weight sharing","Schema-validated artifacts enable federated LLM tool routing","Cross-model federation via typed artifacts: 2pt loss","No weights shared: typed compendiums federate tool knowledge"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The formal convergence guarantee rests on the routing operator being a contraction (Lipschitz constant below 1), a premise the paper neither measures nor proves and that it concedes fails for one of its five test distributions.","fun_headline_variants_meta":{"raw":{"variants":["Typed artifacts let LLMs share tools without sharing weights","Federated tool routing via typed compendiums, no weight sharing","Schema-validated artifacts enable federated LLM tool routing","Cross-model federation via typed artifacts: 2pt loss","No weights shared: typed compendiums federate tool knowledge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000178,"raw_usage":{"total_tokens":1119,"prompt_tokens":719,"completion_tokens":400,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":314}},"tokens_in":463,"tokens_out":400,"duration_ms":4154,"temperature":1.0,"reasoning_tokens":314,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:50:41.750670+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the routing operator's round-to-round Lipschitz constant on real client-update trajectories; if it reaches 1 on any distribution, the Banach fixed-point argument in Theorem A.2 cannot deliver almost-sure convergence. Alternatively, run the cross-family transfer experiment with an LLM family held out during federation; if the accuracy drop consistently exceeds the reported ~2 points, the portability claim is falsified.","supporting_citations":[],"review_version":1}