{"id":"f4096627-fa25-4bc6-ac80-9700b9fd8da4","arxiv_id":"2504.13462","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Stratify improves federated learning on non-IID data by balancing label exposure through a stratified label schedule and label-aware client selection.","lead":"Stratify is a federated learning method that schedules training labels in a balanced order and picks only clients that have data for the current label, so the global model sees all classes evenly even when client data is lopsided. A smart generalist might care because uneven client data is a main reason federated learning underperforms, and the paper claims large gains on standard benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central performance claim rests on an unfair protocol comparison: baselines are stripped of their local-epoch mechanisms and forced into Stratify's single-sample or batch-per-iteration scheme, so the reported gains may be artifacts of the evaluation protocol.","rationale":"The reader's weakest_assumption correctly identifies the evaluation protocol as the most load-bearing weakness. The central claim is empirical superiority over state-of-the-art baselines, and the only support for that claim is the accuracy comparison in Tables I and II. Since the baselines are not run in their native protocols, the comparison cannot substantiate the claim. I independently considered the theoretical analysis in Section III-B: it is indeed weak, since the 'unbiased' estimator is unbiased only with respect to a uniform-label reweighted distribution, not the true data distribution, and the variance decomposition is a restatement. However, the empirical protocol issue is more decisive because even if the theory were correct, the experiments would still not establish the claimed advantage over real baselines. The privacy claim is also questionable, but it is secondary to the main performance claim. Therefore, I agree with the reader's REJECT verdict and recommend no change. The concrete test above would settle the protocol concern by running baselines in their standard configurations; if the accuracy gap persists, the concern would be refuted.","tokens_in":17261,"tokens_out":3306,"duration_ms":31069,"concrete_test":"Re-run the comparison in Table II with each baseline in its standard protocol: FedAvg, FedProx, and SCAFFOLD with E=5 local epochs, batch size 64 or 128, and their standard aggregation rules, while keeping Stratify in its batch-data setting; also record the accuracy at the same number of global updates and at convergence. If FedAvg on CIFAR-10 #C=1 or Dirichlet(0.5) reaches the 70–80% accuracy range commonly reported in the FL literature, the claimed superiority in Tables I–II is an artifact of the stripped-down baseline protocol. Additionally, verify the unbiasedness claim by computing the expected SLS gradient against the true global label distribution on CIFAR-10 Dir(0.5) and reporting the bias magnitude.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that Stratify outperforms state-of-the-art methods under non-IID conditions—is supported almost entirely by Tables I and II, where FedAvg, FedProx, SCAFFOLD, and SFL are evaluated under Stratify's fine-grained update protocol: single-sample per iteration (Algorithm 3) or batch-data per iteration (Algorithm 4). These baselines were designed for several local epochs over each client's full local dataset (FedAvg, FedProx, SCAFFOLD) or sequential full-pass training (SFL). Forcing them to update the global model after every sample (or after a single summed gradient over one sample per label) removes their core mechanisms—local drift correction, proximal regularization, control variates, and local dataset utilization—so the comparison measures a modified algorithm, not the published baseline. This is not a minor tuning issue: with batch size 1, FedAvg reduces to a biased stochastic approximation of the server's aggregated gradient on a skewed local sample, and SCAFFOLD's control variates cannot be meaningfully updated. The paper offers no justification that this protocol is a fair or standard evaluation for these baselines, and it is inconsistent with the literature from which the baselines are drawn. Consequently, the headline empirical gains (e.g., CIFAR-10 #C=1: 81.18% vs 11.56% in Table I) may reflect protocol handicap rather than algorithmic superiority. The theoretical argument in Section III-B does not rescue this: it proves unbiasedness with respect to the uniform-label reweighted distribution, not with respect to the true global data distribution, so it does not establish that the SLS estimator is the right target.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Stratify, a federated learning framework that replaces the usual multi-epoch local training protocol with a Stratified Label Schedule (SLS), label-aware client selection, and fine-grained single-sample or batch-data global updates. The authors claim that this balanced exposure to labels yields an unbiased, lower-variance gradient estimator, that a secure client selection protocol based on CKKS homomorphic encryption preserves privacy, and that Stratify matches IID performance, converges faster, and reduces client-side computation relative to FedAvg, FedProx, SCAFFOLD, and SFL on MNIST, CIFAR-10, CIFAR-100, Tiny-ImageNet, COVTYPE, PACS, and Digits-DG. The central evidence is empirical, presented in Tables I and II, where all baselines are evaluated under Stratify's update protocol.","tokens_in":17516,"tokens_out":5317,"duration_ms":46726,"significance":"If the claims were validated, the idea of explicitly balancing label exposure through a stratified schedule could be a useful addition to the federated learning toolbox, especially for extreme label skew, and the paper includes concrete algorithms and ablations for client selection and chunk size. However, the current evaluation does not support the headline claims. The baselines are forced into a protocol they were not designed for, the theoretical argument is largely definitional and does not establish the claimed variance reduction, and the reported numbers are selected optimistically without standard deviations. The privacy contribution is also narrower than stated, since secure aggregation is explicitly left to future work. Strengths of the manuscript include the detailed algorithm pseudocode and the attempt to analyze communication and implementation overhead, but these do not compensate for the load-bearing problems in the comparison and theory.","major_comments":[{"comment":"The comparison is not apples-to-apples: FedAvg, FedProx, SCAFFOLD, and SFL are evaluated under the proposed single-sample or batch-per-iteration protocol, whereas these methods are defined for multiple local epochs over each client's full local dataset. Under batch size 1, SCAFFOLD's control variates cannot be meaningfully updated, FedProx's proximal term degenerates, and FedAvg's aggregation reduces to averaging single-sample stochastic gradients; the reported gaps (e.g., CIFAR-10 #C=1: 81.18% vs 11.56% in Table I) may therefore be artifacts of disabling the baselines' core mechanisms. A fair evaluation must either use the baselines' native protocols or justify why the modified protocol is the correct one.","section":"Section V, Tables I and II"},{"comment":"The 'unbiased and balanced gradient estimator' claim is definitional: if f_l is uniform, the estimator is an equally weighted average over labels, so P(l)=1/|L| by construction. But the standard FL objective is the sample-average empirical risk; an equally weighted label average is unbiased only under balanced class sizes or a deliberately reweighted objective, neither of which is stated. The variance-reduction argument is also incomplete: the law of total variance decomposition does not show that uniform f_l minimizes Var(E[g|l]); classical stratified sampling would call for allocation proportional to stratum sizes and variances (Neyman allocation). The paper should compare against that baseline and state the objective being estimated.","section":"Section III-B"},{"comment":"The reported numbers are selected optimistically: the epoch with highest test accuracy is chosen for every method, and for feature skew the better of uniform and weighted client selection is reported for Stratify (Section V). No standard deviations or number of seeds are given. This can easily explain small margins (e.g., PACS #D=2 single-sample: 95.35% vs 95.61% for SFL). Please report mean ± std over multiple runs under a single pre-specified selection rule.","section":"Section V, Tables I, II, and V"},{"comment":"The efficiency comparison is internally inconsistent. Table III shows Stratify's per-round cost is two to five times larger (e.g., 145s vs 53s for single-sample FedAvg), while the text claims a per-client total of 199s for Stratify vs 575s for FedAvg; these figures do not follow from the table. A valid comparison should report end-to-end training time to a fixed accuracy target, including serialization and model-passing overhead.","section":"Section V-E, Table III"},{"comment":"The privacy claim is stronger than what is implemented. The secure client selection protocol protects label identities behind placeholders, but the server obtains decrypted global counts N(p) for each placeholder, and secure aggregation of gradients is explicitly deferred ('We leave the exploration of security aspects as an important direction for future work' near Algorithm 4). The text should state that only client selection is privacy-preserving, not the full training procedure.","section":"Section IV-A and Algorithm 4"}],"minor_comments":[{"comment":"There is a typo in the Conclusion: 'Fedvg' should be 'FedAvg'.","section":"Section VI"},{"comment":"The claim that 'chunk size 5 results in around 35% reduction in model transfer frequency' should define the baseline for that percentage and clarify how the reduction is computed.","section":"Section V-F, Table V"},{"comment":"The sentence 'The algorithms are trained for a maximum epoch took in their respective IID case' is unclear; please clarify whether E# denotes an epoch or a communication round and how the maximum is determined.","section":"Section V"},{"comment":"The communication overhead formula T=R×Cupdate×ffreq assumes a single client; please clarify how it applies to the sequential single-sample protocol where model passing occurs between clients.","section":"Section IV-B"},{"comment":"The units in Table IV mix seconds and milliseconds in the header and body; please make the units consistent and define the timing conventions clearly.","section":"Table IV"}],"recommendation":"reject","confidential_remarks":"I concur with the stress-test concern: the central performance claim rests on an unfair protocol comparison, and the theoretical section does not establish the claimed variance reduction. Because the headline results and the theory are both load-bearing, the paper would require a substantial redesign of the experiments and a reworked analysis, rather than minor local fixes. This leads me to recommend rejection rather than major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the core idea—a Stratified Label Schedule that dictates which label each client trains on, combined with label-aware client selection and one-sample or small-batch updates—is genuinely new and worth thinking about. Second, the experimental comparison is so stacked that the headline accuracy gains are not credible as evidence of algorithmic superiority.\n\nWhat is actually new: the SLS construction (repeat each label f_l times, shuffle, consume sequentially), the client selection that only picks clients holding the scheduled label, and the masked-label CKKS protocol for doing this without revealing raw labels. The framework is clearly described and the ablation on chunk size and client selection is a nice touch.\n\nThe problem is in Tables I and II. The baselines FedAvg, FedProx, SCAFFOLD, and SFL are evaluated under the same single-sample or batch-per-iteration protocol as Stratify. That is not how those methods are designed to run. FedAvg needs multiple local steps over the client's full dataset; SCAFFOLD's control variates require local epochs to correct drift; FedProx's proximal term assumes multiple local steps. Forcing them to update the global model after every sample or after a single summed gradient over one sample per label removes exactly the mechanisms that make them work. The result, e.g., CIFAR-10 #C=1: 81.18% vs 11.56%, likely reflects a handicapped baseline, not a superior method. The paper never justifies this protocol choice. On top of that, they report the best epoch for each algorithm and no standard deviations, which further inflates confidence.\n\nThe theory in Section III-B is weaker than the paper suggests. The \"unbiased\" estimator is unbiased with respect to the SLS distribution, not the true data distribution. If labels are imbalanced in reality, a uniform label schedule intentionally changes the target distribution, so the variance-reduction claim is about the wrong quantity. The privacy protocol also has a real gap: CKKS is approximate, so encrypted equality checks are not reliable, and the client that decrypts the difference vector learns something about the existing label set. No code or data is provided.\n\nAll of this is fixable in principle: run baselines in their standard settings or at least alongside the constrained protocol, report variance, release code, reframe the unbiasedness claim, and tighten the privacy argument. As written, the central claim does not hold up.\n\nWho is this for? Someone working on non-IID FL might get a useful idea from the SLS concept, but not from the numbers. If this lands in my inbox, I'd send it to reviewers—the idea deserves a serious look—but I'd expect a rejection and a major rewrite.\n\nRecommendation: engage with it as a promising but unproven idea, not as a demonstrated result.","headline":"A genuinely new label-balancing schedule, but the baselines are run in a protocol that strips their mechanisms, so the headline gains are not credible evidence of superiority.","tokens_in":18123,"tokens_out":4659,"would_cite":false,"duration_ms":40545,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Stratify claims that a Stratified Label Schedule, which balances and shuffles label exposure during federated training, restores accuracy close to IID baselines under non-IID data and converges in far fewer rounds.","keywords":["federated learning","non-IID data","stratified label schedule","label-aware client selection","heterogeneous data","label skew","feature skew","homomorphic encryption"],"falsifier":"Run a label-skew experiment, for example CIFAR-10 with one class per client, where FedAvg, FedProx, and SCAFFOLD are allowed their standard multiple local epochs per round and the same total communication budget as Stratify; if one of them matches or exceeds Stratify's accuracy, the claim that balanced label scheduling is what drives the gains is undercut. Alternatively, measure the empirical variance of the aggregated gradient under SLS versus random client sampling; finding no variance reduction would directly contradict the paper's theoretical claim.","tokens_in":16984,"feed_emoji":"⚖️","tokens_out":6852,"duration_ms":59014,"temperature":0.7,"pith_summary":"Federated learning degrades when client data are non-IID because the global model sees labels and features unevenly. Stratify claims to fix the root cause rather than patch the aggregation step: the server builds a Stratified Label Schedule that repeats each label a controlled number of times, selects only clients holding the scheduled label, and applies fine-grained updates one sample or small batch at a time. The paper argues this produces an unbiased gradient estimator with lower variance, and reports accuracy close to the IID baseline across label-skew and feature-skew benchmarks while converging in many fewer rounds. A sympathetic reader would care because it offers a way to make federated learning robust to extreme heterogeneity without asking clients to share data.","feed_headline":"Balanced label schedules close non-IID federated learning gaps","feed_subtitle":"Stratify schedules every label evenly and reports IID-level accuracy in a fraction of the training rounds.","key_machinery":"The central mechanism is the Stratified Label Schedule (SLS): a shuffled list in which every label $l$ appears $f_l$ times, with $f_l$ either uniform or proportional to a capped global label count, so the global model is exposed to all classes at controlled frequencies. Around it the paper places two supporting mechanisms: label-aware client selection, which restricts each training step to clients whose local data contain the scheduled label, and a fine-grained update scheme that applies one sample or a small batch per global update instead of multiple local epochs. The theoretical claim is carried by a variance-decomposition identity: writing the aggregated gradient as $\\hat{g} = \\frac{1}{|\\mathrm{SLS}|} \\sum_{l \\in \\mathrm{SLS}} g_l$, balanced repetition makes the per-label contribution probability $P(l) = f_l / \\sum_l f_l$ uniform, so $\\mathbb{E}[\\hat{g}]$ is unbiased and the cross-label variance term $\\mathrm{Var}(\\mathbb{E}[\\hat{g} \\mid l])$ is minimized.","core_discovery":"On the paper's own terms, the discovery is that the imbalance that breaks federated learning is not primarily in the aggregation rule but in the model's exposure: when a global model is trained in rounds where each client optimizes its own skewed local data, the aggregate update drifts. Stratify's response is to turn training into a server-orchestrated schedule of labels. The SLS is built by repeating each label a controlled number of times and shuffling; at each step the server assigns the next label only to clients that actually hold that label, and the model is updated immediately on that label's data rather than after full local epochs. The paper claims this yields an unbiased gradient estimate, reduces the between-label variance term of the law of total variance, and lets each label contribute equally to aggregated updates. Empirically it reports that Stratify reaches accuracy comparable to its own IID runs on MNIST, CIFAR-10, CIFAR-100, Tiny-ImageNet, COVTYPE, PACS, and Digits-DG, often in a small fraction of the baselines' rounds.","pith_inferences":["The authors leave implicit that the SLS mechanism does not depend on homomorphic encryption: any protocol that yields aggregate label counts, including trusted-server aggregation, would preserve the core balancing effect; the encryption layer is an implementation choice.","A testable extension would apply the same label-scheduling idea to centralized or asynchronous training under class imbalance, where a server already knows global label frequencies and no privacy machinery is needed.","The single-sample sequential mode is effectively a continual-learning protocol, so a natural extension is a head-to-head comparison with replay-based continual-learning methods under distribution shift; the paper does not evaluate that regime."],"forward_implications":["Under label skew as severe as one class per client, Stratify reports accuracy comparable to the IID setting, while FedAvg, FedProx, SCAFFOLD, and SFL degrade sharply.","Because the schedule controls what the global model sees and when, the method converges in far fewer rounds, for example 8 rounds on CIFAR-10 batch learning versus 50 for FedAvg, lowering cumulative local training time per client.","The label-aware selection mechanism lets clients with only a few samples contribute meaningfully, since each global update needs only data for the scheduled label.","For feature skew, weighted client selection prevents one large-domain client from dominating later iterations, while uniform selection preserves label privacy.","With Batch Normalization, Stratify aggregates sum-based batch statistics rather than averaging local statistics, avoiding the mismatch that hurts FedTAN-style BN in non-IID settings."],"supporting_citations":[{"why":"supplies the FedAvg baseline and its IID assumption that Stratify is designed to replace","marker":"[15]"},{"why":"supplies the FedProx baseline whose proximal regularization is compared against SLS-guided updates","marker":"[10]"},{"why":"supplies the SCAFFOLD baseline whose control variates are compared for correcting local drift","marker":"[12]"},{"why":"supplies the classical stratified sampling theory that motivates the Stratified Label Schedule","marker":"[16]"},{"why":"supplies the variance decomposition used to argue balanced label exposure reduces aggregate gradient variance","marker":"[17]"},{"why":"supplies the sequential federated learning structure that Stratify extends with label-aware client selection","marker":"[21]"},{"why":"supplies the FedTAN batch normalization approach that Stratify adapts into its custom BN layer","marker":"[22]"},{"why":"supplies the non-IID data partitioning methodology used in the experiments","marker":"[23]"}],"fun_headline_variants":["Stratify schedules labels to match IID accuracy in FL","Balanced label exposure fixes non-IID federated learning","Label-balanced sampling slashes federated learning rounds","Server-driven label schedule beats non-IID in FL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the baselines can be fairly compared when they are forced into the same single-sample or batch-per-iteration update protocol that Stratify uses, even though FedAvg, FedProx, SCAFFOLD, and SFL were designed for multiple local epochs over each client's full dataset; if that protocol constraint is removed, the reported accuracy gap may shrink or reverse.","fun_headline_variants_meta":{"raw":{"variants":["Stratify schedules labels to match IID accuracy in FL","Balanced label exposure fixes non-IID federated learning","Label-balanced sampling slashes federated learning rounds","Server-driven label schedule beats non-IID in FL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1605,"prompt_tokens":1051,"completion_tokens":554,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":489}},"tokens_in":667,"tokens_out":554,"duration_ms":4835,"temperature":1.0,"reasoning_tokens":489,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:07:21.146699+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a label-skew experiment, for example CIFAR-10 with one class per client, where FedAvg, FedProx, and SCAFFOLD are allowed their standard multiple local epochs per round and the same total communication budget as Stratify; if one of them matches or exceeds Stratify's accuracy, the claim that balanced label scheduling is what drives the gains is undercut. Alternatively, measure the empirical variance of the aggregated gradient under SLS versus random client sampling; finding no variance reduction would directly contradict the paper's theoretical claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the classical stratified sampling theory that motivates the Stratified Label Schedule"},{"cited_title":"Convergence analysis of sequential federated learning on heterogeneous data,","cited_arxiv_id":null,"evidence_quote":"supplies the sequential federated learning structure that Stratify extends with label-aware client selection"},{"cited_title":"Why batch normalization damage federated learning on non-iid data?","cited_arxiv_id":null,"evidence_quote":"supplies the FedTAN batch normalization approach that Stratify adapts into its custom BN layer"}],"review_version":1}