{"id":"6cab16cf-373e-4c0a-a985-47cf24626f27","arxiv_id":"2610.21348","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"FedSkillGit uses Git-like commits to evolve a shared natural-language skill document across privacy-constrained clients, outperforming baselines by up to 22% and nearly matching centralized performance.","lead":"FedSkillGit is a new way for distributed AI agents to share what they learn without sharing raw data. It borrows the version-control idea of commits to exchange only small, privacy-filtered patches, making collaboration more efficient and auditable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"FedSkillGit's headline aggregates rest on nine single-run cells with no reported variance, and the tightest cells separate methods by 1.6 and 0.8 points; the abstract's \"up to 22%\" is a relative gain in one cell over a self-constructed baseline.","rationale":"The reader's verdict (CONDITIONAL, HIGH confidence) and its two stated grounds are reasonable, and I would keep the verdict where it is. Where I differ is emphasis. The reader's weakest_assumption targets the reliability of the LLM optimizer and merge agents (Appendices A.B and A.C). That is a real mechanism, but it is partly self-controlling for the comparative claim: both FedSkillGit and DocMerge use \"a strong optimizer\" and the same merge machinery where an aggregation step exists (Appendix C-d), so a weak merge agent degrades the treatment and the closest baseline together, and Table II already ablates the experience document's two roles. The more load-bearing gap is one level up, in the evidence base for the aggregate statistics that the strongest_claim is built from: nine single-run cells, no seed sweep, no intervals, with the decisive comparisons at 0.8–1.6 points in two cells. A single run cannot distinguish a 1.6-point gain from run-to-run variation in an LLM-agent loop, yet the paper's language asserts a strict nine-for-nine ordering, and the abstract's \"up to 22%\" is a relative percentage over one cell of a baseline the authors built rather than the published federated method they themselves identify as closest in Related Work II-C. Credit where due: the arithmetic is internally consistent (I verified the mean +4.3, the median -0.8, the -0.8 to -5.1 Centralized range, and the +0.8 to +7.2 DocMerge range against Table I); the family averages that qualify the Centralized comparison are disclosed in b141 rather than hidden; the limitations appendix is unusually candid on the single-document trade, the validation-cost assumption, payload-level privacy, server visibility, and population growth cost; and the no-skill reference in every row is a useful control. Nothing here indicates an error, and none of it justifies REJECT. My recommendation is therefore UNCHANGED, with one condition worth adding to the existing CONDITIONAL: state the seed count and report per-cell dispersion for at least the two tightest cells, and either define the 22% figure in the body or scope the SOTA language to the constructed baselines. If the multi-seed gaps hold strictly positive, my concern does not land and the paper's current wording stands as written.","tokens_in":17015,"tokens_out":12651,"duration_ms":136442,"concrete_test":"Rerun the GPT-5.4-mini condition on SearchQA and LiveMath (the two tightest cells) with at least five independent seeds, resampling the client partition, the participation schedule, and LLM sampling, holding the seed document and per-round budget fixed. For each cell report the mean and a paired interval for FedSkillGit-minus-Independent and FedSkillGit-minus-DocMerge. Decision rule: if any interval for FedSkillGit-minus-Independent includes zero, the \"exceeds Independent in all nine cells\" and \"mean gain of +4.3\" statements must be restated with uncertainty; if any FedSkillGit-minus-DocMerge interval includes zero, the \"trails in all nine cells\" claim and the 22% figure lose their basis. If all intervals stay strictly positive, the concern does not land and the current wording is defensible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has three legs: (i) FedSkillGit beats Independent in all nine cells by a mean of +4.3, (ii) it lands within a median -0.8 of Centralized, (iii) raw client data never leaves. Leg (iii) is honest about its own bounds (b258–b262 acknowledge payload-only enforcement, server-visible commits, and no inference guarantee), so I set it aside per the acknowledgement rule. Legs (i) and (ii) are arithmetically correct: I recomputed all nine FedSkillGit-minus-Independent gaps (2.5, 1.6, 5.7, 2.3, 4.0, 1.6, 10.1, 8.8, 1.7; mean 4.26) and all nine minus-Centralized gaps (median -0.8, mean -1.0). The load-bearing problem is inferential, not numerical. Appendix C reports a single run per cell with one fixed client partition, one fixed participation schedule, and one seed document; no variance, seed count, or interval is reported. Two of the four comparisons that carry the narrative are decided by sub-2-point differences: GPT-5.4-mini/LiveMath has FedSkillGit 28.8, Centralized 28.8, DocMerge 28.0, Independent 27.2, and GPT-5.4-mini/SearchQA has 78.6 vs 77.5 vs 76.1. At that spacing, \"exceeds Independent in all nine cells\" (b139) and \"DocMerge trails in all nine cells\" (b142) assert a strict ordering the design cannot yet certify, because the uncontrolled variance is run-level (sampling non-determinism of executor, optimizer, and merge agents, plus schedule order), and pairing on items within a run does not remove it. Relatedly, the abstract and contribution bullet advertise \"outperforms state-of-the-art baselines by up to 22%\"; no percentage is derived anywhere in the body, and the figure is reconstructible only as the Gemini3-flash/LiveMath relative gain over DocMerge (40.0 vs 32.8 = +22.0%), a self-constructed single-shot-merge baseline, while the one published federated skill-evolution system, FederatedSkill [42], is not run.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper proposes FedSkillGit, a federated skill evolution framework for LLM agents. Instead of transmitting full skill documents, clients submit bounded commits (patches plus local experience) that pass a two-stage privacy filter. The server deduplicates submitted patches, updates a global experience document, and merges surviving patches into a candidate skill, which is published only if it improves on a population-level validation score. Experiments on three datasets and three model families compare FedSkillGit against Independent (each client evolves its own copy), Centralized, No Skill, and DocMerge (single-shot merge). The authors report that FedSkillGit outperforms Independent by a mean gain of 4.3 percentage points and lands within a median gap of -0.8 points of Centralized, without sharing raw client data.","tokens_in":17545,"tokens_out":2762,"duration_ms":28667,"significance":"If the reported gains are statistically reliable, FedSkillGit introduces a principled way to federate the evolution of natural-language skill documents while preserving privacy at the payload level. The idea of using version-control-inspired commits rather than full-document uploads is novel in the federated LLM-agent setting, and the incorporation of a global experience document to guide both local optimization and merging is a plausible mechanism to reduce redundant exploration. The problem of collaborative skill evolution without sharing trajectories is practically relevant for multi-tenant deployments where data owners cannot pool raw data.","major_comments":[{"comment":"All experimental results are based on a single run per cell (one fixed client partition, one fixed participation schedule, one seed document; see Appendix C). No variance, standard deviation, or confidence interval is reported. The central claims—that FedSkillGit 'exceeds Independent in all nine cells' and 'lands within a median gap of -0.8'—are stated as deterministic comparisons. Given the inherent non-determinism of LLM-based executor, optimizer, and merge agents, single runs cannot certify ordering, especially for margins as small as 0.8–1.6 percentage points (e.g., GPT-5.4-mini, LiveMath: 28.8 vs 28.8; SearchQA: 78.6 vs 77.5). The authors should report results from multiple independent runs (at least 5) and provide standard deviations or confidence intervals. Without this, the empirical support for the main conclusions is insufficient.","section":"Section IV and Appendix C","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The abstract and contribution bullet claim that FedSkillGit 'outperforms state-of-the-art baselines by up to 22%'. I was unable to locate a 22% relative improvement in Table I. The largest absolute gain over Independent is 10.1 points (Gemini3-flash, SearchQA: 87.3 vs 77.2), which is a 13.1% relative improvement. The 22% claim appears unsupported. Please specify the exact baseline and cell that yields this number, or correct the claim.","section":"Abstract and Contribution (b9, b139)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper discusses FederatedSkill [42] as the most closely related prior work on federated skill evolution but does not include it as an experimental baseline. While the authors motivate the difference in objective (personalized vs. shared skill), a comparison under a shared-skill variant or at least a discussion of why it cannot be included would strengthen the claim of outperforming 'state-of-the-art baselines'. As it stands, the evaluated baselines (Independent, DocMerge, No Skill) are self-constructed.","section":"Section II-C and IV-A","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The phrase 'state-of-the-art baselines' is imprecise. The paper compares only against self-constructed baselines; suggest rephrasing to 'naturally constructed baselines' or specifying the baselines explicitly.","section":"Abstract and Conclusion","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The text mentions that 'by family average it leads on GPT-5.4-mini by +0.4', but these family averages are not shown in the table. Consider adding a row for the average per model family.","section":"Table I","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"These figures report only convergence rounds. To assess scalability meaningfully, the final task success rate should also be reported for each configuration, to confirm that performance is maintained as the number of clients or participation ratio varies.","section":"Figures 2 and 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The symbol Match(v, c) is defined but not used elsewhere. Clarify that Match is an exact string-matching function (e.g., substring match).","section":"Eq. (7)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The prompts assume that LLMs will reliably follow detailed instructions. Given known prompt sensitivity, the authors should note that the results may depend on prompt wording and that a prompt-sensitivity analysis would strengthen robustness.","section":"Appendix A","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The authors honestly acknowledge that privacy is enforced on the payload, not against inference. This is appropriate. However, the privacy ablation (Table III) measures only first-pass audit rate, not actual information leakage. Consider adding a small adversarial analysis or stating this as a limitation more explicitly in the main text (not just the appendix).","section":"b258–b262","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This paper proposes FedSkillGit, a protocol that replaces full-document uploads with rule-level commits, two-stage privacy filtering, and a global experience document. The motivation is solid: standard federated learning is a bad fit for natural-language skill artifacts, and version-control concepts are a natural alternative. The authors spell out their design choices clearly, run ablations, and include an unusually candid limitations appendix. Credit where due: they know what they didn't solve, and they say it.\n\nThe soft spot is inferential, not conceptual. The headline claims — outperforming independent evolution by 4.3 points on average, landing within 0.8 of centralized — are arithmetically correct but supported by exactly one run per cell (Appendix C). Several cells are decided by gaps of 1.6 and 0.8 points. At that resolution, a single seed partition, one fixed participation schedule, and the non-determinism of the LLM callers mean I cannot tell whether these ordering are real. The abstract's 'up to 22%' is a relative gain over DocMerge, a self-constructed single-shot baseline; the one published federated-skill baseline (FederatedSkill) is discussed but not run. That's a missing comparison that matters.\n\nThe privacy mechanism remains heuristic (the authors acknowledge this: payload-only enforcement, no inference guarantee), which is fine for a systems paper but should be framed less prominently. The code is not shipped, and the runs are not reproducible beyond the description.\n\nWho this is for: researchers building federated agentic systems who want a concrete protocol to start from. It is a useful template, not a final result. A serious referee should see this paper — the idea is new enough and the design is careful — but should ask for variance estimates, more runs, and at least a comparison to FederatedSkill on the same setting. I would accept peer review for this paper conditional on major revision.","headline":"A well-designed federated skill-evolution protocol with honest limitations, but the evidence for its central claim rests on single-run cells without variance — worth reading, but needs stronger empirical grounding before publication.","tokens_in":18110,"tokens_out":982,"duration_ms":15859},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"FedSkillGit enables federated evolution of a shared agent skill via version-control commits, outperforming independent evolution by a mean of 4.3 percentage points and landing within 0.8 points of centralized evolution without sharing…","keywords":["federated learning","skill evolution","LLM agents","version control","commit-based aggregation","privacy filtering","natural-language skills","experience reuse"],"falsifier":"Run the same federated protocol with controlled low-quality optimizer or merge agents (e.g., deliberately noisy, overly conservative, or prone to hallucinations) and measure whether the median gap to centralized evolution widens beyond, say, 5 percentage points; if so, the framework's performance is critically dependent on LLM-agent reliability rather than the commit-merge abstraction itself.","tokens_in":18519,"feed_emoji":"🧩","tokens_out":4647,"duration_ms":50418,"temperature":0.7,"pith_summary":"FedSkillGit is a framework for federated skill evolution that replaces full-document updates with fine-grained, privacy-filtered commits. Each client submits a bounded set of rule-level patches together with a local experience summary, both sanitized and audited before leaving the client. The server deduplicates, accumulates global experience to guide future revisions, merges surviving patches, and validates the candidate skill against client-held validation sets before publishing. If the approach works as claimed, it allows populations of LLM agents to pool their distributed learning without pooling their raw data, matching most of the benefit of centralized evolution across three datasets and three model families.","feed_headline":"FedSkillGit beats independent skill evolution by 4.3 points, nears centralized","feed_subtitle":"Version-control commits enable federated agent-skill collaboration without raw data sharing","key_machinery":"The central mechanism is the commit-based skill update with two supporting structures: a global experience document and a two-stage privacy filter. A commit packages a bounded set of rule-level edits (append, insert-after, replace) together with a local experience summary. The global experience document accumulates failure patterns and rejected revisions from all prior rounds, is read only by the optimizer and merge agents, and reduces redundant exploration. The privacy filter first sanitizes free-text fields by replacing likely sensitive values with typed placeholders, then audits the result against a client's frozen private-value set, returning failed commits to the sanitizer for revision.","core_discovery":"The paper claims that exchanging structured commits rather than full documents solves three core problems of federated skill evolution: privacy leakage (commits are sanitized and audited), communication efficiency (only patches are transmitted), and attribution (each commit is traceable to its proposing client). The key experimental finding is that FedSkillGit outperforms independent evolution in all nine tested cells across SearchQA, LiveMath, and DocVQA using GPT-5.4-mini, Qwen3.5-flash, and Gemini3-flash, with gains between 1.6 and 10.1 percentage points (mean gain +4.3) and a median gap of −0.8 percentage points relative to centralized evolution, demonstrating that federated collaboration recovers most of the benefit of having all data in one place without sharing raw execution trajectories.","pith_inferences":["The commit abstraction could extend beyond skill documents to any structured artifact agents maintain collaboratively—such as prompt libraries, tool definitions, or configuration files—where revisions are localized and semantically meaningful.","Because the publication gate validates against the union of client validation sets, an implicit assumption is that the validation data collectively covers the task distribution well; a client with a too-small or too-biased validation slice could cause harmful revisions to pass or good ones to be rejected.","The two-stage privacy filter addresses content leakage but not inference leakage (as the authors note); combining FedSkillGit with differential privacy or secure aggregation would be a natural next step, though it would need to preserve the structured-natural-language update abstraction."],"forward_implications":["Collaborative skill evolution can be deployed in privacy-sensitive multi-agent settings where raw execution data cannot be pooled, using the same LLM as both executor and optimizer.","The version-control abstraction (commits, merge, validation gate) provides a natural framework for auditing: which client contributed each revision and whether it persisted through validation is recorded by construction.","Convergence cost is predictable, rising with client count (5 rounds at 10 clients to 9 at 100) and falling with participation ratio (13 rounds at 10% to 3 at 50%), enabling deployment planning.","The global experience document reduces redundant knowledge exploration both locally and at the server, as shown by ablation: removing either role drops task success rate from 78.6% to 76.9% or 77.2% on SearchQA with GPT-5.4-mini.","Single-shot merging of full documents is not sufficient: the iterative DocMerge baseline trails FedSkillGit in all nine cells by between 0.8 and 7.2 percentage points."],"supporting_citations":[{"why":"Provides the foundational federated learning paradigm whose full-document aggregation the paper adapts and contrasts with.","marker":"[4]"},{"why":"Direct baseline on federated skill evolution with a personalized-library design; the paper compares against it and makes the opposite design choice of evolving a single shared document.","marker":"[42]"},{"why":"Motivates single-owner skill evolution from execution experience, establishing the centralized-data assumption that federated collaboration challenges.","marker":"[1]"}],"fun_headline_variants":["Federated skill evolution via commit patches, no raw data shared","Patch-based skill updates beat independent learning by 4.3 points","Git-style commits solve privacy, comms, attribution in federated skills","FedSkillGit closes gap to centralized skill evolution without data pooling","Version control for agent skills: client submits patches, not full docs"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The LLM-based optimizer and merge agent (prompted in Appendix A) can reliably extract, generalize, and aggregate useful skill revisions from local execution trajectories and the global experience document without introducing systematic bias, hallucinating invalid patches, or failing to identify redundancies.","fun_headline_variants_meta":{"raw":{"variants":["Federated skill evolution via commit patches, no raw data shared","Patch-based skill updates beat independent learning by 4.3 points","Git-style commits solve privacy, comms, attribution in federated skills","FedSkillGit closes gap to centralized skill evolution without data pooling","Version control for agent skills: client submits patches, not full docs"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":8e-05,"raw_usage":{"cost":7.972356e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":7.972356e-05,"upstream_inference_prompt_cost":4.40412e-05,"upstream_inference_completions_cost":3.568236e-05},"total_tokens":1377,"prompt_tokens":980,"completion_tokens":397,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":305}},"tokens_in":980,"tokens_out":397,"cost_source":"openrouter_usage","duration_ms":5327,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":305,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-16T15:19:45.225003+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run the same federated protocol with controlled low-quality optimizer or merge agents (e.g., deliberately noisy, overly conservative, or prone to hallucinations) and measure whether the median gap to centralized evolution widens beyond, say, 5 percentage points; if so, the framework's performance is critically dependent on LLM-agent reliability rather than the commit-merge abstraction itself.","supporting_citations":[{"cited_title":"Communication-efficient learning of deep networks from decen- tralized data,","cited_arxiv_id":null,"evidence_quote":"Provides the foundational federated learning paradigm whose full-document aggregation the paper adapts and contrasts with."},{"cited_title":"FederatedSkill: Federated Learning for Agentic Skill Evolution","cited_arxiv_id":"2606.03143","evidence_quote":"Direct baseline on federated skill evolution with a personalized-library design; the paper compares against it and makes the opposite design choice of evolving a single shared document."},{"cited_title":"AutoSkill: Experience-Driven Lifelong Learning via Skill Self-Evolution","cited_arxiv_id":null,"evidence_quote":"Motivates single-owner skill evolution from execution experience, establishing the centralized-data assumption that federated collaboration challenges."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}