{"id":"83d57df7-e2c7-4d99-a517-121b8307d58a","arxiv_id":"2412.14226","paper_version":2,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FedSTaS merges FedSTS-style stratified client sampling with FedSampling-style private data-level sampling and reports accuracy gains over FedSTS on MNIST and CIFAR-100 under non-IID splits.","lead":"FedSTaS combines two existing federated learning sampling ideas, client stratification with compressed gradients and local data sampling, into one method. The paper reports accuracy gains over the FedSTS baseline on MNIST and CIFAR-100, especially for non-IID data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 3's data-sampling step is internally inconsistent (fixed size η/m vs per-point probability η/ñ), so the evaluated method is not reproducible and the reported accuracy gains are not traceable to a well-defined algorithm.","rationale":"The paper's central empirical claim is that FedSTaS beats FedSTS in fixed rounds on the reported benchmarks. For that claim to be meaningful, the algorithm that produced the numbers must be well-defined. I identified a concrete internal inconsistency in the data-level sampling step: Algorithm 3 specifies a fixed per-client sample size of η/m but simultaneously gives a per-point probability of η/ñ, which cannot both be true. The prose in Section 2.4 describes a uniform Bernoulli-style scheme, but the pseudocode contradicts it. This is more load-bearing than the imported theoretical lemmas the reader flagged, because the unspecified sampling scheme directly determines the local training data and therefore the measured accuracies. Without code, a reimplementation cannot know which variant was used, so the reported improvements cannot be verified or reproduced. The reader's weakest_assumption focused on the data-level sampling probabilities and their theoretical grounding; my concern is a sharper version at the specification level. I do not think this invalidates the idea—the method could work under either interpretation—but it requires an author clarification or code release before the empirical claim is acceptable. The reader's CONDITIONAL verdict already captures this need for clarification, so my read does not move the verdict; I keep UNCHANGED.","tokens_in":7149,"tokens_out":10594,"duration_ms":97035,"concrete_test":"Obtain the authors' code and inspect the data-sampling implementation in Algorithm 3 / ClientUpdate. If code is unavailable, independently implement both readings—(i) fixed per-client sample of exactly η/m points and (ii) independent Bernoulli(η/ñ) sampling of each point—under the stated hyperparameters (q=0.1, nSGD=3, η=0.01, B=128, T=99, H=10, m=10) on MNIST α=0.01. Determine which implementation reproduces the reported FedSTaS final accuracy of 56.00% and whether the FedSTS baseline of 54.2% is matched under the same settings; also report how clients with n_k < η/m are handled in the code. If neither interpretation reproduces the reported numbers, the empirical claim cannot be attributed to the published algorithm.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.4 says the server 'wishes to sample η observations across the m sampled clients' and, without privacy, 'sample the data uniformly ... with sampling probabilities equal to η/n' (n = total participating data). This implies Bernoulli-style sampling of every data point with probability η/n. However, Algorithm 3 (ClientUpdate) instructs: 'ξ_k^t ← a sample of (η/m) observations from D_k taken with sampling probabilities p_kj = η/ñ'. A fixed-size sample of size η/m cannot have per-point inclusion probability η/ñ: the expected sample size per client would be n_k·η/ñ, not η/m, and for any client with n_k < η/m the fixed-size instruction is undefined under sampling without replacement. If the code instead implements independent Bernoulli(η/ñ) sampling, then the total sampled data is random and the per-client sample sizes vary; either way, the two readings yield different local training sets and thus different test accuracies. The paper does not state which interpretation was used, and code/data are not provided. Since the headline comparison (FedSTaS 56.00% vs FedSTS 54.2% on MNIST α=0.01; 61.90% vs 44.3% at α=0.001; 23.21% vs 18.0% on CIFAR-100) depends on this underspecified choice, the central claim is not reproducible from the text.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FedSTaS, a federated learning client-selection and data-sampling method that combines FedSTS-style client stratification (based on compressed gradients and Neyman allocation) with FedSampling-style local data sampling under local differential privacy. The authors claim unbiasedness and an epsilon-LDP guarantee for the private data-size estimator, and they report higher test accuracy than FedSTS on MNIST and CIFAR-100 under non-IID data. The theoretical results are presented as direct consequences or slight modifications of prior work, and the experimental evidence consists of single-run accuracy curves and bar charts.","tokens_in":7450,"tokens_out":4553,"duration_ms":42497,"significance":"If the central claim is correct, the proposed integration is a reasonable engineering contribution: it combines two known sampling techniques and adds a privacy-preserving data-size estimator with a concrete epsilon=3 calibration. The manuscript is clearly written and makes its algorithmic steps explicit, and the privacy mechanism is well specified. However, the contribution rests primarily on the empirical demonstration, and that demonstration is not yet reproducible or statistically supported: the data-sampling step in Algorithm 3 is internally inconsistent, no code or data are released, and the reported results lack multiple seeds or error bars. The theoretical lemmas are imported from FedSTS and FedSampling without verification that their assumptions hold for the combined scheme. These issues currently prevent the paper from being accepted as a reliable advance.","major_comments":[{"comment":"The data-sampling instruction is internally inconsistent. Section 2.4 says the server wishes to sample eta observations across the m sampled clients with uniform sampling probabilities eta/n, but Algorithm 3 instructs each client to take 'a sample of (eta/m) observations from D_k taken with sampling probabilities p_kj = eta/n-tilde'. A fixed-size sample of size eta/m cannot have per-point inclusion probability eta/n-tilde in general: the expected number of sampled points from client k under Bernoulli(eta/n-tilde) sampling is n_k * eta/n-tilde, not eta/m, and for any client with n_k < eta/m, a fixed-size sample of size eta/m is not even defined under sampling without replacement. The two plausible implementations (fixed-size per-client samples versus Bernoulli per-point sampling) lead to different local training sets and therefore different test accuracies. Since the headline comparisons in Section 3 depend on this underspecified choice, the evaluated method is not reproducible from the text, and no code or data are provided to resolve the ambiguity.","section":"§2.4, Algorithm 3"},{"comment":"The experimental claims are based on single runs without any measure of variability. The text reports point estimates such as 56.00% versus 54.2% on MNIST (alpha=0.01) and 61.90% versus 44.3% on MNIST (alpha=0.001), but no standard deviations, multiple seeds, or statistical tests are reported. Given that the claimed gains include a 1.8-percentage-point difference on one setting, it is essential to show that the improvement is not within run-to-run noise. The authors should provide multi-seed results with error bars and, ideally, a per-seed table.","section":"§3.2, Figures 2–3"},{"comment":"The theoretical results are asserted as immediate consequences or slight modifications of results in FedSTS and FedSampling, but no derivations are shown for the combined FedSTaS scheme. Lemma 2.1 assumes FedSTS's unbiasedness transfers exactly when client sampling is followed by data-level sampling, Lemma 2.2 assumes the FedSampling mean-square-error argument carries over to the stratified client subset, and Lemma 2.4 assumes the LDP argument is unchanged when the randomized response mechanism is combined with the client-stratification procedure. Because the combined sampling distribution is different from either prior method, these are not automatic; the paper should either provide proofs or state explicitly which conditions need to be verified.","section":"§2.5, Lemmas 2.1–2.4"},{"comment":"There is an internal inconsistency in the reporting of the CIFAR-100 experiment. The text says 'after 99 rounds' for the bottom chart, while the Figure 3 caption says 'CIFAR-100 with alpha=0.001 at 199 rounds' and Section 3.1 states T=199 for CIFAR-100. This needs to be corrected, as the number of rounds directly affects the claimed improvement over FedSTS.","section":"§3.2.2, Figure 3 caption"}],"minor_comments":[{"comment":"The phrase 'improving upon FedSTS by 1.8%' should specify whether this is an absolute percentage-point difference or a relative improvement; 56.00% versus 54.2% is a 1.8-percentage-point difference but a relative improvement of about 3.3%.","section":"§3.2, MNIST (alpha=0.01)"},{"comment":"The phrase 'surpassing FedSTS by a remarkable 17.6%' similarly needs clarification: this is a 17.6-percentage-point absolute difference, not a 17.6% relative improvement (the relative gain is about 39.7%).","section":"§3.2.2, MNIST (alpha=0.001)"},{"comment":"The symbol eta is used both for the learning rate in the local update (Algorithm 3, line 5) and for the total data-level sample size in Section 2.4 and Algorithm 2. This makes the hyperparameter list in Section 3.1 confusing, where eta=0.01 appears alongside q=0.1 as the factor determining the sample size. The authors should use distinct symbols (for example, eta for the learning rate and S or n_sample for the requested data sample size).","section":"§2.4 and Algorithm 3"},{"comment":"The definition of n as the total number of participating observations is introduced but then replaced by the private estimate n-tilde; it would help to explicitly state that n is the non-private ideal quantity and n-tilde is its private estimate, and to note that in Algorithm 2 the same symbol eta is reused for both the data sample size and the learning rate.","section":"§2.4"}],"recommendation":"major_revision","confidential_remarks":"The contribution is incremental and depends heavily on FedSTS and FedSampling; the theoretical section imports results from those papers without new derivations. The main evidence is therefore the empirical comparison, which is currently under-supported because of the underspecified sampling step and the absence of code, seeds, and error bars. If the authors resolve the Algorithm 3 ambiguity and supply reproducible multi-seed experiments, the central claim may become defensible; as it stands, the paper is not ready for acceptance. I would also suggest that the editor ask for a clear statement of which implementation of the data-sampling step (fixed-size per client or Bernoulli per point) was used in the reported experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: FedSTaS is exactly what it says—FedSTS client stratification plus FedSampling-style data-level sampling—and on the three benchmark settings it reports consistently higher accuracy than FedSTS. The combination is a legitimate engineering extension, and the authors are upfront that the theory is inherited. But the paper as written has a load-bearing bug in Algorithm 3: Section 2.4 says the server samples η observations across the m participating clients, i.e., per-point probability η/n (or η/ñ under privacy); Algorithm 3 then tells each client to draw a fixed-size sample of η/m observations with per-point probability η/ñ. Those two instructions are not the same. For a fixed-size draw of size η/m, the inclusion probability cannot be η/ñ unless each client has exactly the same n_k; for a Bernoulli draw, the total sample size is random and the per-client sizes vary. The text never says which interpretation was implemented, and there is no code or data. So the headline numbers—56.00 vs 54.2, 61.90 vs 44.3, 23.21 vs 18.0—aren't traceable to a well-defined method. That's the main problem.\n\nWhat's good: the stratification-plus-data-sampling pipeline is natural and worth exploring; the privacy machinery for estimating total sample size with LDP is imported from FedSampling and looks appropriate; the experiments cover three non-IID settings with consistent directional gains. The paper also explicitly lists variance comparison and convergence-rate proofs as future work, which is the right scope.\n\nThe other weaknesses are familiar: single-run, seedless bar charts; no error bars; the \"1.8%\" improvement is 1.8 percentage points, not percent; no comparison to FedSampling alone, so you can't tell how much the data-level sampling adds over FedSTS.\n\nMy take: the idea is fine, the evidence is not. The inconsistency in Algorithm 3 is fixable with a clear definition and released code, and the experimental reporting needs multiple seeds. I'd send it to review because the combination is plausible and the community would benefit from a corrected comparison. But I wouldn't trust the current numbers.","headline":"A reasonable engineering combo undone by an inconsistent data-sampling spec; the reported gains are not traceable to a well-defined algorithm.","tokens_in":7962,"tokens_out":2706,"would_cite":false,"duration_ms":23667,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that FedSTaS, which adds a privacy-preserving per-client data sampling step to the stratified client selection of FedSTS, achieves higher test accuracy than FedSTS alone within a fixed number of training rounds on non-IID…","keywords":["federated learning","client sampling","stratified sampling","data-level sampling","local differential privacy","non-IID data","Neyman allocation","gradient compression"],"falsifier":"Run FedSTaS on MNIST with $\\alpha = 0.01$ for 99 rounds with the stated hyperparameters across multiple random seeds: if its test accuracy does not consistently exceed FedSTS's 54.2%, the paper's central empirical claim fails. Alternatively, record the realized number of data samples per round when each client samples with probability $\\eta/\\tilde{n}$; if that realized total deviates substantially from $\\eta$ across rounds, the centralized-learning mimicry assumption breaks.","tokens_in":6947,"feed_emoji":"🎯","tokens_out":5843,"duration_ms":49348,"temperature":0.7,"pith_summary":"FedSTaS is a federated learning sampling method that combines two ideas: stratified client selection based on compressed gradients with optimal Neyman allocation, and per-client data-level uniform sampling driven by a differentially private estimate of the total participating data size. The paper claims that this two-level scheme reaches higher test accuracy than FedSTS alone within a fixed number of training rounds on both IID and non-IID data, and that the private data-size estimate satisfies $\\epsilon$-local differential privacy. If correct, this gives a practical way to get faster convergence and better final accuracy in heterogeneous federated learning while keeping client data sizes private.","feed_headline":"FedSTaS beats FedSTS accuracy on non-IID federated benchmarks","feed_subtitle":"Adding per-client data sampling to stratified client selection raises final test accuracy within a fixed round budget.","key_machinery":"The central machinery is a two-stage sampling pipeline. At the client level, FedSTaS uses the FedSTS mechanism: clients are clustered by their Information Squeezed (compressed) gradient vectors into $H$ strata, the $m$ sampled clients are allocated across strata in proportion to $N_h S_h$ where $S_h$ is the stratum standard deviation (Neyman allocation), and clients within a stratum are sampled with probability proportional to gradient norm. At the data level, each participating client computes a clipped local size $n_{hi,c} = \\min(n_{hi}, M-1)$, draws a fake size from a uniform multinomial distribution, and applies a Bernoulli mix with parameter $\\alpha$ to produce a perturbed size $r_{hi}$; the server aggregates these into the private total estimate $\\tilde{n}$ and uses sampling probability $\\eta/\\tilde{n}$ for each client's local data. The paper's theoretical claims rest on two inherited lemmas: that the client-level sampling is unbiased, and that the private estimator $\\tilde{n}$ converges in mean square to the true total and satisfies $\\epsilon$-LDP under the stated choice of $\\alpha$.","core_discovery":"FedSTaS is a two-level sampling scheme. At the client level, the server stratifies clients by their compressed gradients, re-allocates the client sample size across strata using optimal Neyman allocation, and samples clients within each stratum with probability proportional to gradient norm. At the data level, each participating client computes a clipped local data size, mixes it with a fake size via a Bernoulli draw, and the server forms a private estimate of the total participating data size, $\\tilde{n} = (R - (1-\\alpha)M m)/\\alpha$, which is then used as the denominator in a uniform per-client data sampling probability $\\eta/\\tilde{n}$. The paper claims three theoretical properties: client-level unbiasedness inherited from FedSTS, mean-square convergence of the data-sampling probability to the centralized probability as the number of sampled clients grows, and $\\epsilon$-LDP for the private size estimate when $\\alpha = (e^\\epsilon - 1)/(e^\\epsilon + M - 2)$. Empirically, the paper reports FedSTaS reaches 56.00% on MNIST ($\\alpha=0.01$), 61.90% on MNIST ($\\alpha=0.001$), and 23.21% on CIFAR-100 ($\\alpha=0.001$), exceeding FedSTS's 54.2%, 44.3%, and 18.0% within the same number of rounds.","pith_inferences":["Because the paper only compares against FedSTS, a natural testable extension is whether adding data-level sampling to other client-selection baselines, such as clustered or regularized sampling, yields similar gains; the data-level mechanism is not obviously tied to stratification.","The paper does not establish end-to-end privacy: the $\\epsilon$-LDP result covers only the local data-size estimate, while the compressed gradients themselves are transmitted without a DP guarantee, so a full privacy-accounting extension would be needed for practical private deployment.","The realized total number of data samples per round is only approximately $\\eta$, since each participating client samples with probability $\\eta/\\tilde{n}$ independently; a direct measurement of the realized sample size would reveal how closely the centralized-learning mimicry holds in practice."],"forward_implications":["Within a fixed round budget, FedSTaS should reach higher test accuracy than FedSTS on non-IID MNIST and CIFAR-100 splits, with the gap widening as data heterogeneity intensifies.","The private total-size estimator lets the server run centralized-style data sampling without learning any client's exact data size, with a formal $\\epsilon$-LDP guarantee.","The DP variant (DP+FedSTaS) should retain most of the accuracy benefit while providing a privacy guarantee of $\\epsilon = 3$.","The client-level sampling scheme inherits unbiasedness from FedSTS, so the aggregated update does not systematically drift from the full-client aggregation.","As the number of sampled clients grows, the data-level sampling probability approaches the centralized learning probability, so the method behaves more like centralized learning in the large-$m$ limit."],"supporting_citations":[{"why":"Supplies the client stratification, compressed-gradient clustering, Neyman allocation, and importance sampling that FedSTaS inherits at the client level, and is the baseline for the experimental accuracy comparison.","marker":"(Gao et al., 2024)"},{"why":"Supplies the data-level uniform sampling and the privacy-preserving total-size estimator whose theoretical lemmas FedSTaS adapts for the participating clients.","marker":"(Qi et al., 2023)"},{"why":"Defines the FedAvg aggregation and the unbiased client-sampling framework in which FedSTaS is formulated and evaluated.","marker":"(McMahan et al., 2023)"}],"fun_headline_variants":["FedSTaS combines client and data sampling to beat FedSTS","Client and data sampling boost federated accuracy over FedSTS","FedSTaS: stratified client sampling plus data sampling wins","Two-level sampling lifts federated model accuracy past FedSTS","FedSTaS uses dual sampling to surpass FedSTS on benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the unbiasedness, convergence, and privacy guarantees proved separately for the client-sampling and data-sampling components continue to hold exactly when the two components are combined as in FedSTaS, even though the total sampled data size is only approximately $\\eta$.","fun_headline_variants_meta":{"raw":{"variants":["FedSTaS combines client and data sampling to beat FedSTS","Client and data sampling boost federated accuracy over FedSTS","FedSTaS: stratified client sampling plus data sampling wins","Two-level sampling lifts federated model accuracy past FedSTS","FedSTaS uses dual sampling to surpass FedSTS on benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1485,"prompt_tokens":999,"completion_tokens":486,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":403}},"tokens_in":615,"tokens_out":486,"duration_ms":4455,"temperature":1.0,"reasoning_tokens":403,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:32:28.471322+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run FedSTaS on MNIST with $\\alpha = 0.01$ for 99 rounds with the stated hyperparameters across multiple random seeds: if its test accuracy does not consistently exceed FedSTS's 54.2%, the paper's central empirical claim fails. Alternatively, record the realized number of data samples per round when each client samples with probability $\\eta/\\tilde{n}$; if that realized total deviates substantially from $\\eta$ across rounds, the centralized-learning mimicry assumption breaks.","supporting_citations":[{"cited_title":"FedSTS : A Stratified Client Selection Framework for Consistently Fast Federated Learning","cited_arxiv_id":null,"evidence_quote":"Supplies the client stratification, compressed-gradient clustering, Neyman allocation, and importance sampling that FedSTaS inherits at the client level, and is the baseline for the experimental accuracy comparison."}],"review_version":1}