{"id":"4824e330-b57d-450e-a0fe-79e4dfd863ce","arxiv_id":"2412.08138","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LeaDQ uses QMIX-based multi-agent reinforcement learning to learn per-client policies for selecting which streaming unlabeled samples to query labels for in federated learning, achieving higher global model accuracy than several baselines.","lead":"The paper introduces LeaDQ, a reinforcement-learning method that helps each client in federated learning decide which newly arriving, unlabeled data samples to spend labeling budget on. It matters because labeling data is expensive, and better sample selection can improve the shared model while keeping all client data private.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LeaDQ's reward (Eq. 10) requires ground-truth labels for a held-out set at the server; those labels are never charged to the method's annotation budget, so the reported gains over baselines may reflect extra labeled data rather than a better query policy.","rationale":"The paper's proposal is internally coherent and the experiments are broad, but the most load-bearing condition for the central claim is the labeled held-out set. Without ground-truth labels for D_held, Eq. 10 cannot be computed and the MARL policies have no reward signal; the method's learning mechanism depends on exactly the resource the problem setting declares scarce. The reader correctly identified this as the weakest assumption. I add a sharper point: even when such a held-out set is available, the experimental comparison is not label-budget-matched because LeaDQ consumes 1000 extra labeled samples while the baselines do not. This makes the headline 'outperforms benchmarking algorithms' questionable as a statement about querying efficiency rather than about access to additional supervision. The proposed budget-matched ablation would settle whether the advantage persists under equal label cost. Since the reader already issued a CONDITIONAL verdict and the concern reinforces rather than overturns it, the verdict is unchanged.","tokens_in":13865,"tokens_out":8332,"duration_ms":97284,"concrete_test":"Run a label-budget-matched comparison on Table 4 (quantity skew, SVHN) and Table 8 (50 clients): either (a) grant each baseline the same 1000 extra labeled samples used as D_held, for instance as an initial labeled set, or (b) deduct 1000 labels from LeaDQ's query budget, for instance by running 1000/Nq fewer query rounds. If LeaDQ's accuracy advantage disappears or falls below the reported margins, the outperformance claim is conditional on an uncharged held-out labeling cost.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 defines the RL reward as R^r = Acc(theta^r; D_held) - Acc(theta^{r-1}; D_held), which requires ground-truth labels for D_held. The paper's motivating problem is streaming unlabeled data with expensive annotation; it states the D_held assumption in Section 5.1 but never discusses the labeling cost of D_held or counts it in the comparison. Experiments use D_held of 1000 samples (Table 6) for LeaDQ, while the uncertainty, coreset, LoGo, and KAFAL baselines receive no analogous labeled validation data. The central claim that LeaDQ 'outperforms the benchmarking algorithms' is therefore not established under equal annotation budget: the policy training signal itself is purchased with labels that the problem setting treats as scarce. A secondary consequence is that the policy is optimized on one specific D_held, and no experiment varies D_held size or composition, so overfitting to that set is unexplored.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies active data querying in federated learning when unlabeled samples arrive at clients as a data stream. It formulates the querying process as a Dec-POMDP and proposes LeaDQ, a QMIX-based multi-agent reinforcement learning approach in which each client's local policy selects N_q samples per round from its arrived unlabeled stream, while a server-side mixing network coordinates the local Q-functions using a global state derived from a held-out dataset. The reward is the one-round change in held-out accuracy. Experiments on SVHN, CIFAR-100, 20Newsgroup, and Tiny-ImageNet compare LeaDQ with Uncertainty, Coreset, LoGo, and KAFAL, reporting consistent accuracy improvements.","tokens_in":14040,"tokens_out":4609,"duration_ms":47354,"significance":"If the claims hold, LeaDQ is a useful step toward principled, learned querying policies for streaming federated active learning. The paper provides a clean problem formulation, a reasonable CTDE instantiation via QMIX, a public code release, a motivating experiment, and evaluations across image and text tasks with non-IID data. The central weakness is that the policy-training signal in Eq. (10) requires ground-truth labels for a server-held dataset of size 1000 (Table 6), and this annotation cost is never charged to LeaDQ's budget nor given to the baselines. In addition, the empirical evidence rests on three-seed averages without error bars or significance tests, and one state-of-the-art baseline (KAFAL) collapses in two of the reported settings without explanation. These issues currently prevent the paper from establishing its headline claim of outperforming the benchmarking algorithms under an equal annotation budget.","major_comments":[{"comment":"The reward R^r = Acc(θ^r; D_held) − Acc(θ^{r−1}; D_held) explicitly requires ground-truth labels for the held-out dataset D_held, whose size is listed as 1000 in Table 6. The server uses these labels to train the query policy, while the uncertainty, coreset, LoGo, and KAFAL baselines receive no analogous labeled validation set. Because the paper's motivating problem treats labels as expensive and scarce, the reported accuracy gains may come from the extra labeled data used only by LeaDQ rather than from the learned querying policy itself. Please either remove the label requirement (e.g., use an unlabeled proxy for the reward), add an equal-annotation-budget comparison in which the baselines also receive the same labeled held-out data for their own tuning, or explicitly justify why the D_held labels should not be counted. In addition, no experiment varies the size or composition of D_held, so overfitting of the policy to that particular held-out set is unexplored.","section":"Section 5.1, Eq. (10), Table 6"},{"comment":"The main results are averaged over three random seeds with no error bars, standard deviations, or significance tests. Several of the reported margins are extremely small or even negative: in Table 3 (α=1.0) LeaDQ is 66.50% versus KAFAL's 66.48%, and in Table 2 (Nq/Nu=2/10) LeaDQ is 76.28% versus KAFAL's 76.32%. Without per-seed results or a statistical comparison, the claim that LeaDQ consistently outperforms the baselines is not supported, especially for the later-round curves in Fig. 4 where the plotted lines appear close.","section":"Appendix B.2, Fig. 4, Tables 2–4"},{"comment":"KAFAL, a state-of-the-art federated active learning baseline, collapses to roughly 20% accuracy in the quantity-skew setting (Table 4) and on Tiny-ImageNet (Table 7), while it performs competitively elsewhere (e.g., 64.67% in Table 3 with α=0.5). The paper does not explain this collapse or provide any diagnostic, such as sensitivity to hyperparameters or a check of whether the implementation is appropriate for these setups. If KAFAL is not properly configured for these scenarios, the comparison is unfair; if it genuinely fails, that is an interesting finding that should be analyzed. Please add an explanation or ablation, or remove the affected rows with justification.","section":"Table 4 and Table 7"}],"minor_comments":[{"comment":"There are typos: '20Newsgrpup' in the Fig. 4 subplot label, 'dsitribution' in Section 6.2, and 'Global Corset' in the Fig. 3 caption should be 'Global Coreset'.","section":"Fig. 4, Section 6.2"},{"comment":"The third column header of Table 5 reads 'SVHN' but the column reports the DistilBERT model and should be labeled '20Newsgroup'.","section":"Table 5"},{"comment":"The sentence 'LeCun et al. 1998.' is missing its closing parenthesis; it should be 'LeCun et al. 1998)'.","section":"Appendix B.1"},{"comment":"The row 'Total arrived samples in Table 1' is confusing: Table 1 is the dataset summary, not the implementation-detail table. Please renumber or reword this row so it refers to the correct table.","section":"Table 5"},{"comment":"The Dec-POMDP tuple is written as ⟨S, A, P, R, O, K, γ⟩, which omits the observation function and the number of agents in the standard notation. This is a minor formal clarity issue, but the tuple should be defined consistently with the text that follows.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The central idea is interesting and the experiments are broad, but the uncharged labeled held-out dataset is a serious fairness issue that must be resolved before the main claim can be accepted. I would also ask the authors to add statistical rigor and to explain the KAFAL collapse. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"LeaDQ is a serious attempt at a real problem: when unlabeled data arrive in streams at FL clients, how to decide which samples to label. The framing as a Dec-POMDP and the use of QMIX to learn local policies with a global reward is new in this setting, and the paper is mostly clear. The experiments are extensive — SVHN, CIFAR-100, 20Newsgroup, Tiny-ImageNet, varying skew and client count — and the code is available. On the face of it, LeaDQ consistently beats uncertainty, coreset, LoGo, and KAFAL. That is worth refereeing.\n\nThe largest soft spot is the server's labeled held-out set. The global state (Eq. 8) is the model's confidence on D_held, and the reward (Eq. 10) is the accuracy delta between rounds on that same set. The paper states this in Section 5.1, but never charges the labels of D_held against the annotation budget, and the baselines receive no analogous labeled validation data. So part of the reported gain may be purchased with labels that the problem setting treats as scarce. That is a load-bearing unfairness in the comparison, not a fatal flaw, but it needs to be addressed — either by using an unlabeled proxy for the reward, or by giving the baselines the same budget and reporting the total label cost.\n\nA few secondary issues: results are averaged over three seeds with no error bars or significance tests, so we cannot tell whether gains are real or noise. The reward itself is a one-round accuracy delta on a fixed 1000-sample held-out set, which is noisy; no sensitivity to D_held size or composition is reported. KAFAL collapses under quantity skew (Table 4), which suggests a possible implementation issue — the paper doesn't comment on it. And the claim in the motivating example that LeaDQ 'achieves comparable performance' to Global Coreset is made without any variance information.\n\nThe framing and the algorithm are still the main contribution, and the work is clearly on the right track. The citation pattern is fair; the related work covers both federated active learning and RL-based AL. I'd send this to a serious referee, but with a request for a careful look at the held-out-label issue and a statistical comparison. If those are fixed, this would be a credible contribution to the federated active learning subfield.","headline":"Streaming federated active learning cast as a Dec-POMDP with QMIX; a solid contribution whose evaluation is undermined by an un-costed labeled held-out set at the server.","tokens_in":14607,"tokens_out":2468,"would_cite":true,"duration_ms":25545,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LeaDQ treats federated data querying as a multi-agent decision problem and learns local selection policies that improve global model accuracy beyond current federated active learning baselines.","keywords":["federated learning","active learning","data streams","multi-agent reinforcement learning","QMIX","Dec-POMDP","non-IID data","data querying"],"falsifier":"Run the paper's SVHN setup with a mismatched held-out set (for example, CIFAR-100 in place of SVHN); if the global state and reward lose all information about the target distribution, the proposed method should collapse to baseline-level accuracy if the central claim about learning from global signals is right.","tokens_in":13638,"feed_emoji":"🎯","tokens_out":8121,"duration_ms":75357,"temperature":0.7,"pith_summary":"This paper targets a gap in federated learning: client data arrive as unlabeled streams, labeling is expensive, and the samples that help one client's local model are not necessarily the samples that help the global model. The authors claim that data querying in this setting should be framed as a collaborative decentralized decision problem rather than a local heuristic. They propose LeaDQ, a multi-agent reinforcement learning algorithm that trains each client's local sample-selection policy against a global reward: the improvement in the federated model's accuracy on a held-out set. Experiments on SVHN, CIFAR-100, 20Newsgroup, and Tiny-ImageNet show LeaDQ consistently beating uncertainty sampling, coreset selection, LoGo, and KAFAL. If the claim holds, federated systems can spend their labeling budget on samples that actually advance the global model, not just each client's local view.","feed_headline":"Federated clients learn which unlabeled samples to label","feed_subtitle":"By training local query policies against global accuracy, LeaDQ beats four active-learning baselines on image and text streams.","key_machinery":"The load-bearing mechanism is the Dec-POMDP/QMIX combination. A Dec-POMDP formalizes the fact that each client chooses which samples to query under partial observation—only its own logits—while the objective is joint. QMIX provides the training-time bridge between local policies and the global goal: the joint Q-value is Qtot = ψ(Q1, ..., QK, sr), a monotonic function of the local Q-values conditioned on the global state sr, and the network is updated by a temporal-difference loss against the shared held-out accuracy reward. This is what lets the algorithm train locally executable policies against a signal no single client can see, without ever moving raw data off the clients.","core_discovery":"The paper's central discovery is that stream-based federated active learning can be cast as a Dec-POMDP in which every client is an agent. Each agent's local observation is the current global model's predictive logits on the newly arrived unlabeled samples, its action is the choice of which Nq samples to label, and all agents share the reward Rr = Acc(θr; Dheld) − Acc(θr−1; Dheld), the change in held-out accuracy after training on the newly queried labels. Training uses the QMIX factorization: a monotonic mixing network combines local Q-values with a global state (the model's maximum prediction confidence on the held-out set) into a joint Q-value, so each client can learn a decentralized policy that is nevertheless optimized for the global objective. The accompanying motivating experiment shows where the benefit comes from: a Global Coreset that coordinates selections across clients clearly beats Local Coreset, and LeaDQ approaches the global oracle's accuracy while keeping data decentralized.","pith_inferences":["The method's practical reach depends on the labeled held-out set; a deployment without a representative D_held would need a proxy signal (for example, confidence calibrated on a small validation slice or a self-supervised estimate), and that substitution is the most direct testable extension.","Since the reward is measured after full FedAvg rounds, policy training is expensive; computing the reward on a small validation batch or every few rounds would make the approach more plausible in bandwidth-limited settings.","The QMIX monotonicity constraint limits the interaction structure among clients; richer factorizations could capture cases where one client's querying decision should negatively influence another's, which the current formulation rules out."],"forward_implications":["In stream-based federated active learning, a learned query policy can beat both generic active-learning heuristics (uncertainty, coreset) and federated-specific baselines (LoGo, KAFAL) under non-IID client data.","Because execution only requires the current model's logits and the local policy network, the querying decisions stay decentralized and compatible with FedAvg-style privacy constraints.","The method remains effective when the number of clients grows to 50 and when data heterogeneity is varied in both distribution-based and quantity-based ways.","The learned policies select samples whose label and feature distributions resemble the target global distribution rather than the local client distribution, which is the mechanism behind the accuracy gains.","The framework can be adapted to different arrival volumes and query budgets by changing the action constraint |a| = Nq, and its performance is stable across episode-length hyperparameters."],"supporting_citations":[{"why":"Supplies QMIX, the monotonic value-factorization algorithm used to train the local query policies against the joint reward.","marker":"Rashid et al. 2018"},{"why":"Supplies FedAvg, the federated optimization procedure that the global model follows between query rounds.","marker":"McMahan et al. 2017"},{"why":"Defines coreset active learning, used both as a baseline and as the Local/Global Coreset motivating comparison.","marker":"Sener and Savarese 2018"},{"why":"Defines uncertainty sampling, the baseline whose predictive-logits idea is adopted for the local observation.","marker":"Wang and Shang 2014"},{"why":"Defines LoGo, a federated active learning baseline that LeaDQ is compared against.","marker":"Kim et al. 2023"},{"why":"Defines KAFAL, the knowledge-aware federated active learning baseline that LeaDQ is compared against.","marker":"Cao et al. 2023"},{"why":"Provides the Dec-POMDP formalism used to frame decentralized data querying in federated learning.","marker":"Oliehoek and Amato 2016"},{"why":"Provides the Dirichlet-based data partitioning used to simulate non-IID client distributions.","marker":"Li et al. 2022"}],"fun_headline_variants":["LeaDQ: federated agents learn to pick labels from streams","Multi-agent RL decides which samples to label in federated learning","Smart query policy for federated active learning on data streams","Global reward guides local labeling in federated streams"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the server has a labeled held-out dataset that follows the target distribution, because that dataset is the only source of both the global state signal and the reward used to train the querying policies; without it, there is no learning signal.","fun_headline_variants_meta":{"raw":{"variants":["LeaDQ: federated agents learn to pick labels from streams","Multi-agent RL decides which samples to label in federated learning","Smart query policy for federated active learning on data streams","Global reward guides local labeling in federated streams"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00039,"raw_usage":{"total_tokens":2044,"prompt_tokens":925,"completion_tokens":1119,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":1051}},"tokens_in":541,"tokens_out":1119,"duration_ms":11937,"temperature":1.0,"reasoning_tokens":1051,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:09:33.599112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's SVHN setup with a mismatched held-out set (for example, CIFAR-100 in place of SVHN); if the global state and reward lose all information about the target distribution, the proposed method should collapse to baseline-level accuracy if the central claim about learning from global signals is right.","supporting_citations":[{"cited_title":"S.; Farquhar, G.; Foerster, J","cited_arxiv_id":null,"evidence_quote":"Supplies QMIX, the monotonic value-factorization algorithm used to train the local query policies against the joint reward."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies FedAvg, the federated optimization procedure that the global model follows between query rounds."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines coreset active learning, used both as a baseline and as the Local/Global Coreset motivating comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines uncertainty sampling, the baseline whose predictive-logits idea is adopted for the local observation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines LoGo, a federated active learning baseline that LeaDQ is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines KAFAL, the knowledge-aware federated active learning baseline that LeaDQ is compared against."},{"cited_title":"A.; and Amato, C","cited_arxiv_id":null,"evidence_quote":"Provides the Dec-POMDP formalism used to frame decentralized data querying in federated learning."}],"review_version":1}