{"id":"687a66b4-b034-4bb2-8f9f-6ec7883ac230","arxiv_id":"2506.07440","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fed-ICL iteratively refines QA answers via federated in-context learning with only label transmission, showing convergence on a linear attention model and gains on MMLU and TruthfulQA.","lead":"This paper proposes Fed-ICL, a federated learning method for question answering where clients never send model parameters or raw data, only predicted labels. It iteratively refines answers by having clients use the server's current predictions as in-context examples for their local language models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.1's fixed point is (2I-H_cont)^{-1}w_limit, not w_limit; the claimed convergence to the centralized ICL optimum holds only when H_cont=I, which fails almost surely for finite samples even in-distribution.","rationale":"This concern is load-bearing because it targets the paper's central theoretical contribution. The recurrence and corollary are formally correct, but the paper's interpretation of the fixed point as the centralized optimal answer is not. The reader's strongest claim already noted the H_cont=I condition; my test sharpens this by showing that even in the theorem's own finite-sample setting, H_cont does not equal I almost surely, so the claimed guarantee is not merely limited by real-world heterogeneity. I do not escalate to reject because the algorithm is still well-defined and empirically motivated, and the theoretical section can be repaired by replacing the optimality claim with an explicit H_cont=I assumption or by redefining the target as w*. However, the current formulation of the convergence guarantee is inaccurate, and the paper should be revised accordingly. I therefore keep the reader's CONDITIONAL verdict.","tokens_in":24048,"tokens_out":17027,"duration_ms":219179,"concrete_test":"Run a one-dimensional instantiation of Theorem 4.1 with Gamma=1, L=1, N=2, M=1: take local covariates x_1=1, x_2=2, query x_q=0.5, and arbitrary labels y. Then H_cont = ((1^2+2^2)/2)*(0.25)=0.625, so w* = w_limit/(2-0.625) approximately 0.727 w_limit. Iterating w_{k+1}=0.5 H_cont w_k + 0.5 w_limit converges to w*, and the query prediction differs from w_limit^T x_q, directly contradicting the optimality claim. To generalize, sample random Gaussian covariates with Lambda_client=Lambda_server=Lambda and compute H_cont for N=M=10, verifying that ||w*-w_limit||>0 with probability one.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing problem is internal to the paper's own simplified model. In Theorem 4.1, the update is w_{k+1}=1/2 H_cont w_k + 1/2 w_limit, and the text identifies w_limit^T x as 'the linear function learned for query x through ICL with example datasets D_1,...,D_L' (Eq. 18), i.e., the centralized all-client answer. The fixed point of the recurrence is w*=(2I-H_cont)^{-1}w_limit, not w_limit. These coincide only if H_cont=I. With finite N and M, H_cont = Gamma^{-1}(sum_{i,n} x_i^n (x_i^n)^T/(NL)) Gamma^{-1}(sum_m x_m x_m^T/M) is a product of sample covariance matrices and equals I only by coincidence, so it fails with probability one under the paper's Gaussian assumptions. Thus Corollary 4.2 proves convergence to a possibly query-dependent biased weight, not to the claimed optimal answer. The 'H_cont approx I' paragraph in Section 4.1 is an asymptotic, distribution-aligned statement; it does not rescue the finite-sample theorem. Additionally, the condition in Corollary 4.2 should be ||H_cont||_2<2 rather than <=2, since equality can make (2I-H_cont) singular.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces Fed-ICL, a federated in-context learning framework for question answering in which a server iteratively refines predicted answers by having clients relabel their local data using the current global context and then predict the server's query examples using both original and relabeled local data. The server aggregates client predictions, and the process repeats over several rounds. The paper provides a convergence analysis for a simplified single-layer linear self-attention model and reports experiments on MMLU and TruthfulQA showing accuracy gains over FL and parameter-free baselines at lower communication cost.","tokens_in":24376,"tokens_out":8267,"duration_ms":87836,"significance":"The proposed framework is novel and practically motivated, and the experimental evaluation is comparatively thorough, including ablations, heterogeneity analysis, a privacy analysis, and communication-cost comparisons. The algebraic derivation of the recurrence in Theorem 4.1 is transparent and builds on prior work. However, the headline theoretical claim—that Fed-ICL converges to the centralized ICL optimum—is not actually established by the proved result: the fixed point of the recurrence is (2I-H_cont)^{-1}w_limit, which generally differs from w_limit. With a corrected statement and appropriately scoped claims, the paper would still be useful for its algorithmic contribution and empirical results, but the current theoretical framing overstates what is proven.","major_comments":[{"comment":"The recurrence w_{k+1} = (1/2)H_cont w_k + (1/2)w_limit has fixed point w^* = (2I-H_cont)^{-1}w_limit, not w_limit. The statement in the contributions (Section 1) and conclusion (Section 7) that Fed-ICL 'converges to the optimal answers conditioned on all client datasets' is therefore not supported by Corollary 4.2; it would require H_cont = I, which fails with probability one for finite N and M under the paper's Gaussian assumptions because H_cont is a product of sample covariance matrices. The asymptotic argument in Section 4.1 (H_cont ≈ I when T, N, M → ∞ and the distributions align) does not justify the finite-sample claim. Please restate the guarantee as convergence to w^* and analyze the bias w^* - w_limit, or prove conditions under which w^* = w_limit.","section":"Theorem 4.1 and Corollary 4.2; Section 1 contribution bullet and Section 7 conclusion"},{"comment":"The condition should be ||H_cont||_2 < 2, not ≤ 2. If ||H_cont||_2 = 2, the matrix (2I-H_cont) can be singular and w^* may not be well-defined; moreover, the contraction factor 1/2 ||H_cont||_2 in Eq. (3) equals 1 in that case, so geometric convergence is not guaranteed. The paper also does not provide primitive conditions on N, M, L, d, Λ, Λ_client, and Λ_server under which the spectral-norm assumption holds with high probability; as stated, Corollary 4.2 is conditional on an event whose probability is not analyzed.","section":"Corollary 4.2"},{"comment":"The convergence theorem assumes all clients share the same pretrained single-layer linear self-attention model, Gaussian covariates, and distributional alignment with the pretraining distribution. These restrictions are acknowledged in Remark 4.3, but the abstract, introduction, and conclusion state the convergence-to-optimality claim without these caveats. The claims should be scoped to the simplified model, and the theory should not be described as a guarantee for the practical Fed-ICL setting with heterogeneous LLMs and non-Gaussian text data.","section":"Section 4, 'LM Setup' and Remark 4.3"}],"minor_comments":[{"comment":"The phrase 'sends back to C i k to server' should read 'sends C^i_{k+1} to the server'.","section":"Algorithm 1, line 8"},{"comment":"The proof uses 'episode k' where the algorithm description uses 'round k'; please align the terminology.","section":"Appendix B, proof of Theorem 4.1"},{"comment":"The term 'parameter-free' is used to describe methods that do not transmit model parameters, but Fed-ICL does transmit predicted labels; consider clarifying this terminology to avoid confusion.","section":"Abstract and Section 1"},{"comment":"These ablation plots show a single trajectory per setting without error bars or repeated-seed variance; reporting standard deviations over multiple runs would strengthen the empirical claims.","section":"Figures 5–8"},{"comment":"The use of GPT-4o-generated answers as ground truth for TruthfulQA is non-standard; since the metric then measures agreement with a particular model rather than with human truthfulness judgments, this choice should be justified more carefully.","section":"Section 6.1, Evaluation Metric"}],"recommendation":"major_revision","confidential_remarks":"The main issue is that the paper's advertised theoretical guarantee is not what is actually proven. If the authors correct the fixed-point statement and soften the optimality claim, the paper may be suitable for publication. The experiments are reasonably extensive but would benefit from variance reporting. The related-work coverage is adequate and the concurrent-work disclosure is handled appropriately."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"New here is the Fed-ICL iteration: the server holds predicted labels, clients relabel their local examples using those as ICL context, and the server averages. That loop is new, and the paper also contributes a no-label variant and a kNN filtering trick. The writing credits the concurrent Chen et al. 2025 and the earlier Wu et al. 2024b, so the 'first' claim is appropriately qualified. On the theory side, the recurrence derived for a single-layer LSA model is correct: w_{k+1} = (1/2)H_cont w_k + (1/2)w_limit. The algebra in Theorem 4.1 checks out given the assumptions imported from Zhang et al. 2023a.\n\nThe soft spot is the interpretation of that recurrence. The paper says Fed-ICL converges to the optimal answer conditioned on all client datasets, which is w_limit^T x. But the fixed point is (2I - H_cont)^{-1}w_limit, not w_limit. These coincide only when H_cont = I. With finite N and M, H_cont is a product of sample covariances, so it equals I only by coincidence; under the paper's own Gaussian assumptions it fails with probability one. The 'H_cont approximately I' paragraph is an asymptotic statement and does not rescue the finite-sample theorem. There is also a minor boundary issue: Corollary 4.2 needs ||H_cont||_2 < 2, not <= 2, because equality can make (2I - H_cont) singular. This gap is real but fixable: either prove a bound on the bias or replace 'optimal' with 'converges to a point determined by the sample covariances.'\n\nThe experiments show consistent gains over the baselines and the ablations are informative. But they lack error bars, and several hyperparameters (C, K, temperature, kNN size) appear selected on the same benchmarks. No code or artifacts are released, which makes the empirical claims hard to verify. The privacy section is mostly a screenshot of one prompt-extraction attack with no quantitative metric; it does not support a strong privacy claim.\n\nAll in all, this is a useful algorithm paper with an overclaimed theory section. The proof is not circular and the core iteration is genuinely new, so a serious referee should engage with it. The right fix is honest rewording of the convergence target and a bit more empirical rigor. I'd bring it to a reading group mainly to discuss where the theory overshoots, and I'd cite it as related work on federated ICL if I worked in that area.","headline":"A genuinely new federated-ICL loop with a convergence proof that proves less than claimed; worth engaging, but the 'optimal' language needs revision.","tokens_in":24900,"tokens_out":2094,"would_cite":true,"duration_ms":23992,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Under a linear self-attention idealization, Fed-ICL's round-based label exchange converges to the pooled-data answer and outperforms debate and ensemble baselines on QA benchmarks.","keywords":["federated learning","in-context learning","question answering","iterative refinement","linear self-attention","large language models","parameter-free communication","privacy-preserving learning"],"falsifier":"In a controlled regression experiment with a single-layer linear self-attention model, compute $H_{\\mathrm{cont}}$ and $w_{\\mathrm{limit}}$ from the client data and check whether the observed server weights obey $w_{k+1} = \\frac{1}{2}H_{\\mathrm{cont}}w_k + \\frac{1}{2}w_{\\mathrm{limit}}$ across rounds; any systematic deviation, or a case with $\\|H_{\\mathrm{cont}}\\|_2 > 2$ where the iteration still converges, would falsify the paper's central convergence claim.","tokens_in":23887,"feed_emoji":"🔁","tokens_out":17357,"duration_ms":149280,"temperature":0.7,"pith_summary":"This paper introduces Federated In-Context Learning (Fed-ICL), a round-based scheme in which a server and clients refine answers to common questions without transmitting model weights or raw local data. The central claim is that iterating this refinement converges to a fixed point that coincides with the pooled-data answer when client and query covariances align with the pretraining distribution, at least in the linear self-attention idealization used for the theory. The engine is a simple linear recurrence for the server's implicit weight vector, and the paper proves linear convergence whenever a covariance-based contraction matrix satisfies a spectral bound. Experiments on MMLU and TruthfulQA show the iterative scheme outperforming one-shot ensembling, debate, and federated fine-tuning baselines while communicating only predicted answers. This matters because high-quality in-context examples are exactly what privacy-conscious federated settings cannot share directly.","feed_headline":"Federated ICL converges without sending model weights","feed_subtitle":"Clients exchange only predicted answers; theory and QA benchmarks show iterative refinement keeps improving quality.","key_machinery":"The machinery is the recurrence $w_{k+1} = \\frac{1}{2}H_{\\mathrm{cont}}w_k + \\frac{1}{2}w_{\\mathrm{limit}}$ that emerges from composing two ICL passes inside Algorithm 1. In the first pass each client uses the server's current answers to relabel its own local examples; in the second it uses both its original and relabeled examples to answer the server's queries, and the server averages the resulting labels. For the single-layer linear self-attention model, each pass is a linear readout, and composing the two passes collapses into the affine map above. The matrix $H_{\\mathrm{cont}}$ is the contraction that governs convergence, $w_{\\mathrm{limit}}$ is the pooled-data target, and the bound $\\|H_{\\mathrm{cont}}\\|_2 \\le 2$ is the condition under which repeated application of the map contracts toward the fixed point.","core_discovery":"At the center of the paper is a linear dynamical story about in-context learning. For a single-layer linear self-attention (LSA) model, the paper shows that Fed-ICL's server-side predicted label for every query $x_m$ can be written as $\\hat{y}_{k,m} = w_k^\\top x_m$, and the implicit weight vector evolves as $w_{k+1} = \\frac{1}{2}H_{\\mathrm{cont}}w_k + \\frac{1}{2}w_{\\mathrm{limit}}$. Here $H_{\\mathrm{cont}}$ is built from the empirical covariances of client example covariates and server query covariates, and $w_{\\mathrm{limit}}$ is exactly the weight vector the same LSA model would learn from the union of all client datasets. Theorem 4.1 derives this recurrence, and Corollary 4.2 shows that under $\\|H_{\\mathrm{cont}}\\|_2 \\le 2$ the server weights converge at a linear rate to $w^* = (2I - H_{\\mathrm{cont}})^{-1}w_{\\mathrm{limit}}$. The intended reading is a federated analogue of classic convergence-to-the-global-model guarantees: Fed-ICL reaches pooled-data answer quality without pooling data.","pith_inferences":["Inference: if the contraction intuition carries over to nonlinear LLMs, the framework should converge fastest when each client's local question distribution matches both the server's query distribution and the model's pretraining distribution; this yields a testable client-selection or data-filtering principle.","Inference: the theorem leaves a gap between the fixed point $w^*$ and the pooled-data target $w_{\\mathrm{limit}}$, so in realistic settings converged answers will likely differ from the pooled-data oracle; measuring that gap empirically would quantify the cost of federated instead of centralized ICL.","Inference: the reported privacy evaluation tests prompt extraction from individual client responses, not whether multiple refinement rounds leak more about local datasets over time, so multi-round leakage remains an open empirical question."],"forward_implications":["Federated QA can run by exchanging only predicted labels: per-round communication cost is proportional to the number of queries, not to the model size, while answer quality still improves across rounds.","The convergence target $w_{\\mathrm{limit}}$ is the pooled-data weight vector, so the framework's asymptotic answer quality is capped by what a single model could achieve with all client data together.","The contraction condition ties guaranteed convergence to alignment between client data, server queries, and the pretraining distribution; when that alignment fails, the guarantee does not apply.","Even without ground-truth labels at clients, the Fed-ICL-Free variant builds pseudo-labels from the server's refined answers and still beats debate-style baselines that ignore local datasets.","Empirically, more interaction rounds and semantically filtered local examples improve accuracy, while adding more clients with stronger data heterogeneity hurts it."],"supporting_citations":[{"why":"Supplies the single-layer linear self-attention model and the closed-form ICL prediction formula that Theorem 4.1 derives the Fed-ICL recurrence from.","marker":"Zhang et al. (2023a)"},{"why":"Defines in-context learning over function classes, which is the client-side operation the algorithm repeatedly invokes.","marker":"Garg et al., 2022"},{"why":"Provides FedAvg, the classical aggregation baseline and the average-aggregation analogy behind the server update.","marker":"McMahan et al., 2017"},{"why":"LLM-Debate is the main iterative parameter-free baseline; the paper argues Fed-ICL achieves lower communication cost and better use of local data.","marker":"Du et al., 2023"},{"why":"LLM-Blender contributes the GENFUSER aggregation used for fusing client answers on the TruthfulQA benchmark.","marker":"Jiang et al., 2023"},{"why":"Mixture-of-Agents is one of the parameter-free ensemble baselines the experiments are compared against.","marker":"Wang et al., 2024"},{"why":"Contributes the MMLU benchmark used for the multiple-choice QA experiments.","marker":"Hendrycks et al., 2020"},{"why":"Contributes the TruthfulQA benchmark used for the open-ended answer-generation experiments.","marker":"Lin et al., 2021"}],"fun_headline_variants":["Fed-ICL iterates answers to converge to pooled-data quality","No weights transmitted: Fed-ICL matches global model quality","Linear convergence proof: Fed-ICL reaches pooled-data answer","Clients share only answers; Fed-ICL converges to global","Iterative answer exchange achieves pooled-data performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee assumes every client runs the same simplified single-layer attention model, that client and server questions are Gaussian with covariances matching the pretraining distribution, and that the matrix controlling contraction is not too large in operator norm; outside those conditions the claimed convergence to pooled-data answer quality is not proven.","fun_headline_variants_meta":{"raw":{"variants":["Fed-ICL iterates answers to converge to pooled-data quality","No weights transmitted: Fed-ICL matches global model quality","Linear convergence proof: Fed-ICL reaches pooled-data answer","Clients share only answers; Fed-ICL converges to global","Iterative answer exchange achieves pooled-data performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000616,"raw_usage":{"total_tokens":2880,"prompt_tokens":983,"completion_tokens":1897,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":1813}},"tokens_in":599,"tokens_out":1897,"duration_ms":14841,"temperature":1.0,"reasoning_tokens":1813,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:33:57.075290+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a controlled regression experiment with a single-layer linear self-attention model, compute $H_{\\mathrm{cont}}$ and $w_{\\mathrm{limit}}$ from the client data and check whether the observed server weights obey $w_{k+1} = \\frac{1}{2}H_{\\mathrm{cont}}w_k + \\frac{1}{2}w_{\\mathrm{limit}}$ across rounds; any systematic deviation, or a case with $\\|H_{\\mathrm{cont}}\\|_2 > 2$ where the iteration still converges, would falsify the paper's central convergence claim.","supporting_citations":[],"review_version":1}