{"id":"ba542bef-b7ae-44c9-a468-f4c3069a000b","arxiv_id":"2608.08525","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"DPS-FD, a domain-aware proxy selection method with a vocabulary-constrained LLM proxy generator, is claimed to improve out-of-distribution federated distillation, but its OOD evaluation is compromised by proxy data drawn from the test domains.","lead":"This paper proposes a method called DPS-FD that picks the most relevant proxy samples for federated distillation, plus an LLM-based way to create proxy text when none is available. The authors report higher F1 scores than several baselines on Amazon review sentiment tasks, but the experimental setup leaks the test domains into the training proxy data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported with-proxy OOD gains are contaminated because FD-GP and DPS-FD distill on proxy data drawn from the same unseen domains as the OOD test set; this measures target-domain adaptation, not OOD generalization.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing issue: the with-proxy OOD setup uses proxy data from the test domains, so the headline result does not measure out-of-distribution generalization. I concur with this reading. The paper's method is internally coherent: the selection mechanism is clearly specified, the equations are consistent, and the code release is a positive sign. The problem is not the math but the experimental design. Because Table 1 is the main evidence for the abstract's claim of addressing OOD challenges, the central claim is unsupported. A single rerun with proxy domains disjoint from test domains would be decisive; if the gain persists, the method may still have value as a target-domain adaptation technique, but the OOD framing would remain unjustified for the with-proxy setting. The no-proxy arm is also compromised because the vocabulary constraint uses real proxy data, contradicting the premise of having no proxy data. These issues jointly keep the verdict at REJECT; no verdict change is needed.","tokens_in":14730,"tokens_out":3591,"duration_ms":40588,"concrete_test":"Re-run the with-proxy experiment with proxy data restricted to a held-out set of domains disjoint from the OOD test domains. For example, keep client private domains as automotive, baby, clothing, health, sport, but use proxy data only from beauty, patio, pet, shoes, and software; test on the original five client-unseen domains. If DPS-FD's global F1 advantage over FD-LP and FD-GP largely disappears, the reported 82.9 vs 81.4 result depends on access to target-domain proxy data and is not evidence of OOD generalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that DPS-FD improves OOD generalization under proxy data is not supported by the experimental design. In Section 5.3, FD-GP is defined as using 'global proxy data composed of samples from all domains that are unseen by the clients,' while Section 4.1 defines the OOD test set as 'sampled from the domains not seen by the clients.' Thus the proxy data used in the main with-proxy experiments is drawn from the same domains as the test set. DPS-FD's server-side selection (Eqs. 6-9) explicitly selects top-K proxy samples most similar to the global centroid, so the global model is distilled on unlabeled data from the test distribution. The Table 1 gain (DPS-FD 82.9 vs FD-GP 81.4 global F1) is therefore evidence of target-domain adaptation, not out-of-distribution robustness. The without-proxy arm has a related weakness: Section 4.3 constructs the vocabulary by 'tokenizing real proxy data,' so the no-proxy scenario still depends on real proxy data for the lexical constraint. The proposed selection mechanism is coherent and could be useful, but the benchmark does not establish the advertised OOD claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DPS-FD, a domain-aware proxy selection framework for federated distillation, in which each client and the server select top-K proxy samples most similar to their feature centroids (Eqs. 6-9) to construct a domain-aware proxy set. When no proxy data is available, the authors introduce a vocabulary-constrained LLM-based generation strategy (Section 4.3) that uses a global vocabulary derived from real proxy data. Experiments on Amazon review sentiment classification with heterogeneous clients report improved global F1 over baselines: 82.9% with proxy data and 80.6% without proxy data in Tables 1 and 4.","tokens_in":14934,"tokens_out":4423,"duration_ms":47110,"significance":"The idea of jointly selecting proxy samples at both client and server levels is coherent, and the method is computationally light and clearly described. The release of code and data is commendable. However, the experimental protocol does not measure out-of-distribution generalization as claimed: the with-proxy baseline FD-GP uses proxy data from the same domains as the OOD test set, and the without-proxy pipeline depends on real proxy data for vocabulary construction. As a result, the central advertised claim that DPS-FD addresses OOD distribution shifts is not supported by the evidence presented.","major_comments":[{"comment":"The main with-proxy evaluation does not constitute an OOD test. Section 5.3 defines FD-GP as using \"global proxy data composed of samples from all domains that are unseen by the clients,\" and Section 4.1 defines the OOD test set as \"sampled from the domains not seen by the clients.\" Thus the proxy data in the principal comparison is drawn from the same domains as the test set. DPS-FD then selects top-K proxy samples via Eqs. (6)-(9), and the global model is distilled on unlabeled samples from the test distribution. The reported improvement in Table 1 (DPS-FD 82.9 vs FD-GP 81.4 global F1) therefore measures target-domain adaptation, not out-of-distribution robustness. This invalidates the abstract's claim that the model \"effectively address[es] the challenges of distribution shifts under OOD.\"","section":"5.3, 5.1, 4.1"},{"comment":"The without-proxy pipeline is not self-contained. Section 4.3 states that \"we first construct a vocabulary from tokenizing real proxy data,\" which presupposes access to proxy data even in the no-proxy scenario. If that real proxy data includes the unseen test domains (as in the with-proxy setup), the generated proxy inherits lexical information from the test distribution, so the without-proxy results in Table 1 also fail to establish OOD generalization. The Limitations section does not acknowledge this dependency on real proxy data for the vocabulary constraint.","section":"4.3"},{"comment":"The captions of Tables 1 and 2 report p<0.001, but the manuscript does not state the statistical test used, the number of random seeds, or any variance measure over runs. Without this information, the small reported gaps (e.g., 82.9 vs 81.4 global F1) cannot be assessed for significance, further weakening the support for the central claim.","section":"Tables 1-2, 5.4"}],"minor_comments":[{"comment":"There is a typo in Section 1: \"promblem\" should read \"problem.\"","section":"1"},{"comment":"The phrase \"aVocabulary-Constrained\" in Section 4.3 is missing a space; it should read \"a Vocabulary-Constrained.\"","section":"4.3"},{"comment":"In Section 5.3, \"DPS-FDis\" is missing a space; it should read \"DPS-FD is.\"","section":"5.3"},{"comment":"Figure 3's axis labels appear garbled in the manuscript text, with repeated \"/uni000000...\" strings, which makes the figure caption and the top-K analysis hard to interpret.","section":"Figure 3"}],"recommendation":"reject","confidential_remarks":"The experimental design flaw is fundamental to the paper's main claim. Re-running with a properly separated proxy set (e.g., proxy data only from client domains) would be a substantial new evaluation, and the without-proxy pipeline would still require clarifying how the vocabulary is obtained without real proxy data. Given that the core advertised result is currently unsubstantiated, I recommend rejection rather than major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the central OOD claim doesn't hold as stated. The with-proxy experiments give the server proxy data sampled from the same domains as the OOD test set. Section 5.3 defines FD-GP as \"global proxy data composed of samples from all domains that are unseen by the clients,\" and Section 4.1 defines the OOD test set as domains not seen by the clients. So the global model is distilled on unlabeled data from the test distribution. The reported 82.9 vs 81.4 F1 gain is target-domain adaptation, not out-of-distribution robustness. This is the load-bearing flaw, and it contaminates the main result.\n\nWhat's actually new: the DPS selection rule (top-K cosine similarity to client and global centroids, then union) is clean and computationally cheap. The vocabulary-constrained LLM generation is a reasonable answer to the no-proxy-data problem for NLP, and the analysis in Figure 4 shows the vocabulary constraint does pull generated data closer to real word statistics. Credit also for releasing code.\n\nThe without-proxy arm has a softer but related problem: the vocabulary is built from real proxy data, so \"without proxy\" is not strictly true; it is \"without sharing proxy data for distillation.\" That may be acceptable if clearly stated, but it's not stated.\n\nOther soft spots: DAFKD (Wang et al., 2023a), a closely related domain-aware FD method, is cited but never compared. There are no error bars or repeated trials; the p<0.001 claim needs a statistical method. The top-K parameters appear to be chosen on the test set (Table 3 reports several values but no held-out selection).\n\nWho is this for? Someone working on federated distillation might still find the selection heuristic worth trying in a properly set OOD benchmark. But as submitted, the evidence does not back the advertised claim. I'd want to see a revision with a clean OOD protocol: proxy data drawn from a disjoint distribution (e.g., a broader pool that does not contain test domains, or a leave-one-domain-out setup), and the without-proxy arm either using no real proxy data at all or clearly labeled as vocabulary-only access. That is a fixable paper, and the core idea is not wrong, just mistested.\n\nRecommendation: I would not reject it outright as nonsense, but I would not accept it as is. A serious referee could help the authors redesign the experiments. If the editor's policy is to desk reject papers with load-bearing design flaws, that's defensible. If they are willing to send it out, the review will be straightforward: point at Section 5.3 and ask for the proper control.","headline":"Proxy data in the main OOD experiments is sampled from the same unseen domains as the test set, so the reported gains measure test-domain adaptation, not OOD generalization; the core selection idea is still salvageable.","tokens_in":15512,"tokens_out":3813,"would_cite":false,"duration_ms":37147,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Domain-aware proxy selection lets federated distillation keep its accuracy when test data shifts.","keywords":["federated distillation","out-of-distribution","proxy data selection","domain-aware","LLM-generated proxy data","vocabulary constraints","heterogeneous clients","knowledge distillation"],"falsifier":"Run DPS-FD and FD-GP with proxy data drawn only from the five client domains (automotive, baby, clothing, health, sport) and evaluate on the five held-out domains; if the F1 gap over FD-LP shrinks or vanishes, the reported 82.9% result is an artifact of target-domain leakage rather than evidence of OOD generalization.","tokens_in":14448,"feed_emoji":"🔀","tokens_out":7401,"duration_ms":61934,"temperature":0.7,"pith_summary":"This paper claims that the weak spot in federated distillation under distribution shift is the proxy data used for knowledge transfer, and that fixing it is enough to produce large gains. It proposes DPS-FD, which each round has every client and the server score proxy samples by cosine similarity to their domain centroids, keep the top-K, and distill on the union. When no real proxy data exists, it generates proxy reviews with an LLM constrained to the global vocabulary of real data. On a five-domain Amazon sentiment benchmark with held-out test domains, the method reports 82.9% F1 with proxy data and 80.6% without, beating the FD baselines and the comparison methods DS-FL, MHAT, and FedKD. A sympathetic reading: proxy-data selection and lexical alignment, not new aggregation rules, are what let FD survive domain shift.","feed_headline":"Proxy selection lifts accuracy of federated distillation under shift","feed_subtitle":"Selecting proxy samples by domain similarity gives 82.9% F1 with real proxy and 80.6% with LLM-generated text.","key_machinery":"The load-bearing object is the domain-aware proxy set $D_p^* = \\bigcup_{k} P_k \\cup P_g$, rebuilt each round. Client $k$ picks $P_k$ as the top-$K$ proxy samples with highest cosine similarity between the proxy sample's feature vector and the client's mean feature vector (centroid) over private data; the server picks $P_g$ similarly using the global centroid over proxy data. Distillation then runs on this union. The complementary machinery is the vocabulary-constrained LLM generator: a global vocabulary is built by tokenizing real proxy data, and few-shot prompts force the LLM to produce reviews using only that vocabulary, pulling synthetic text toward the real lexical distribution (shown by TF-IDF projection).","core_discovery":"On its own terms, DPS-FD establishes two things. First, substituting a domain-aware proxy set, the union of each client's top-K proxy samples closest to its private-data centroid and the server's top-K samples closest to the global centroid, for the raw proxy set improves global and local F1 under domain shift while slightly reducing communication cost (98.3% and 95.9% of the standard FD cost). Second, when proxy data is unavailable, proxy text generated by an LLM under a vocabulary constraint drawn from real data is close enough to real proxy data to keep most of the benefit; constrained generation outperforms unconstrained generation by about 2.9 F1 points. The paper also reports that global-domain proxy data helps more than local-domain proxy data, and that DPS-FD lifts the weakest client domain (baby) from 47.6% F1 to 84.6%.","pith_inferences":["Editorial inference: the comparison FD-GP uses global proxy data composed of samples from the very domains that are unseen by the clients, so the strong result may be measuring adaptation to the test domains via proxy leakage rather than generalization to truly unseen domains; a stricter protocol with proxy restricted to client domains would separate the two.","Editorial inference: since selection is by similarity to centroids, a testable extension is replacing the top-K cosine rule with uncertainty- or diversity-aware sampling to see whether the gains come from denoising or from retaining hard samples; the paper itself notes that smaller top-K can remove informative near-boundary samples.","Editorial inference: the same selection-plus-constrained-generation recipe could transfer to other token-based modalities such as code or clinical notes, where domain vocabularies are cheap to construct, though the paper only demonstrates sentiment classification."],"forward_implications":["If proxy selection is the main lever, FD can be made more resistant to domain shift without changing client training or aggregation rules.","Because the selected proxy set is smaller than the full set, communication cost drops while accuracy rises, so the method's benefit does not trade against bandwidth.","The vocabulary-constraint result implies that lexical distribution matching, not just semantic plausibility, is a controllable factor in LLM-generated proxy data.","The global model's held-out accuracy improves with more clients (from 5 to 10), suggesting that broader domain coverage in private data strengthens the global centroid and the selected proxy."],"supporting_citations":[{"why":"Supplies the standard FD method (FD-LP) and the communication-cost baseline that DPS-FD is compared against.","marker":"(Lin et al., 2020)"},{"why":"Introduces proxy-data-based model distillation, the setting DPS-FD modifies with domain-aware selection.","marker":"(Li and Wang, 2019a)"},{"why":"Provides the knowledge-distillation objective (KL divergence on soft labels) used in the global and local distillation steps.","marker":"(Hinton et al., 2015)"},{"why":"DS-FL is one of the SOTA comparison methods in Table 4 that DPS-FD outperforms.","marker":"(Itahara et al., 2023)"},{"why":"MHAT is one of the SOTA comparison methods in Table 4 that DPS-FD outperforms.","marker":"(Hu et al., 2021)"},{"why":"FedKD is one of the SOTA comparison methods in Table 4 that DPS-FD outperforms on global F1.","marker":"(Wu et al., 2022)"},{"why":"Defines the federated learning paradigm that FD and DPS-FD build on.","marker":"(McMahan et al., 2017)"},{"why":"Prior work using public or synthesized proxy data to reduce domain gaps, which motivates DPS-FD's proxy construction.","marker":"(Jeong et al., 2023)"},{"why":"Prior work on adaptive weighting and ensemble strategies for robustness in distillation, a baseline line of defense against distribution shift.","marker":"(Zhu et al., 2021)"}],"fun_headline_variants":["Domain-aware proxies lift federated distillation under shift","OOD federated distillation: pick proxies by domain similarity","Federated distillation gets OOD-ready with domain-aware proxies","LLM-generated proxies close OOD gap for federated distillation","Domain-aware proxy selection boosts FD accuracy under shift"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire OOD claim depends on treating proxy data sampled from the test domains as though it still measures out-of-distribution robustness; if that assumption is dropped, the reported gains may only show that training on target-domain proxy data helps, which is expected.","fun_headline_variants_meta":{"raw":{"variants":["Domain-aware proxies lift federated distillation under shift","OOD federated distillation: pick proxies by domain similarity","Federated distillation gets OOD-ready with domain-aware proxies","LLM-generated proxies close OOD gap for federated distillation","Domain-aware proxy selection boosts FD accuracy under shift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000756,"raw_usage":{"total_tokens":3330,"prompt_tokens":882,"completion_tokens":2448,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":2369}},"tokens_in":498,"tokens_out":2448,"duration_ms":18729,"temperature":1.0,"reasoning_tokens":2369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:33:19.254028+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DPS-FD and FD-GP with proxy data drawn only from the five client domains (automotive, baby, clothing, health, sport) and evaluate on the five held-out domains; if the F1 gap over FD-LP shrinks or vanishes, the reported 82.9% result is an artifact of target-domain leakage rather than evidence of OOD generalization.","supporting_citations":[],"review_version":1}