{"id":"85521f07-1ec5-4a97-a589-6439037fb134","arxiv_id":"2505.15244","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A vertical federated learning algorithm that assigns more important features and larger local models to more reliable NWDAF clients reduces test loss by up to 18% over a dropout-robust baseline in simulated 5G core network settings.","lead":"This paper proposes a reliability-aware feature assignment algorithm for vertical federated learning in 5G core networks, assigning the most informative data features to the most reliable network analytics functions. It reports lower prediction loss than a baseline in simulated DASH video QoE prediction, with larger gains when reliability is lower.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline loss reduction may be inflated by test-set checkpoint selection in Sec. VI-B; no held-out validation split is used, so the claimed 14.5%/18% gains are not yet independently verified.","rationale":"The reader's weakest_assumption points to fixed and accurate reliability forecasts, which is a real but secondary concern: the paper does not test sensitivity to forecast noise, yet the empirical protocol also keeps success probabilities fixed across training and testing. My reading identifies a more immediately falsifiable threat to the central claim: Sec. VI-B selects checkpoints using test performance. This is not a flaw in the proposed algorithm's internal logic, but it undermines the validity of the empirical comparison as evidence of generalization. The same protocol is applied to the baseline, so the comparison is superficially fair, but best-of-run selection on the test set can bias relative gains if the two methods have different variance across rounds. Since the paper reports no confidence intervals, the 14.5% and 18% figures could be inflated by this selection effect. A validation split is a minimal and decisive fix, and the released code should make it straightforward to run. I therefore agree with the reader's CONDITIONAL verdict, but for a different load-bearing reason than the one singled out as their weakest assumption; hence partial agreement.","tokens_in":9136,"tokens_out":3869,"duration_ms":38398,"concrete_test":"Using the released code, split the DASH dataset into train/validation/test (e.g., 70/15/15 or 5-fold cross-validation); select checkpoints for both the proposed and baseline algorithms by validation loss only, then compute held-out weighted test loss per ID pattern and overall, with bootstrap 95% confidence intervals over the same 10 Beta(8,2) and Beta(10,6) runs. If the proposed method still shows roughly 14.5%/18% improvement, the concern is resolved; if the margin shrinks or reverses, the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim rests on comparing 'best-performing' models whose checkpoints are selected by test performance during training. Sec. VI-B states: 'We also continually store the best-performing server and client models in terms of its test performance, computed during each round... Once training concludes, we revert to the top performing models.' This is test-set leakage: the reported weighted test losses in Figs. 1-4 are minima over checkpoints evaluated on the same test data, not the expected loss of a model selected by a principled validation rule. If the proposed method's per-round model trajectory is noisier than the baseline's (plausible, since reliability-scaled embeddings concentrate capacity in a few clients), best-of-run selection can systematically favor the proposed method even when its final or average model is no better. No standard errors or confidence intervals are reported, so the 10-run average cannot separate this selection bias from genuine generalization gains. The checkpoint-selection protocol is therefore load-bearing: without a validation split, the reported 14.5% and 18% reductions are not an unbiased estimate of what the deployed algorithm would deliver.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a reliability-aware feature allocation and model definition algorithm for vertical federated learning (VFL) in a 5G core network. The server uses centralized data orchestration to compute feature importances, then assigns more important features and larger embedding dimensions to clients (NWDAFs) with higher predicted reliability, defining local model sizes under a communication budget. The authors evaluate the method against a VFL baseline from [10] on a DASH QoE dataset with K=4 clients under two reliability scenarios (Beta(8,2) and Beta(10,6)), reporting approximately 14.5% and 18% lower average weighted test loss. The manuscript includes pseudocode for initialization, feature association, training, and inference, and a public code repository.","tokens_in":9336,"tokens_out":7299,"duration_ms":63206,"significance":"The motivating setting—VFL in 5G core networks with centralized data management and NWDAF reliability constraints—is timely, and the proposed mechanism is simple and implementable. The strongest assets are the architectural framing tied to 3GPP NWDAFs, the availability of dataset and code, and a concrete falsifiable claim. However, the paper is an empirical heuristic comparison with no theoretical analysis, and the current evaluation protocol undermines the headline numbers: checkpoints are selected on test performance, the reported reduction is computed with an absolute-value metric, and only 10-run averages without error bars are provided. If these issues are fixed, the contribution would be a useful engineering result; as written, the effect size is not yet established.","major_comments":[{"comment":"The evaluation protocol selects the 'best-performing server and client models in terms of its test performance' during training and reverts to these top-performing models for all reported results. This is test-set leakage: the displayed losses are minima over checkpoints evaluated on the same test data, not the expected loss of a model chosen by a held-out validation rule. If the proposed method's per-round trajectory is noisier than the baseline's (plausible, because reliability-scaled embeddings concentrate capacity in a subset of clients), best-of-run selection can systematically favor it even if the final or average model is no better. Please select checkpoints using a validation split, or report the final-round model and the validation-selected model separately, and report per-run results.","section":"Section VI-B (checkpoint selection)"},{"comment":"The text states that the proposed method achieves 'approximately 14.5% lower test loss' and '~18% reduction', but the displayed formula is Σ_{m=1}^{15} |ℓ_proposed(ID=m) − ℓ_SoTA(ID=m)| / Σ_{m=1}^{15} ℓ_SoTA(ID=m). This is a mean absolute relative deviation, not a signed average reduction. Figs. 2 and 4 show several positive per-ID differences (proposed method worse), so the absolute-value aggregation can overstate the improvement and is inconsistent with the 'lower loss' phrasing. Report the signed average difference (with the distribution across the 10 runs) rather than this absolute-value quantity.","section":"Section VI.C (percentage expression)"},{"comment":"The algorithm relies on each NWDAF providing a reliability forecast that remains fixed throughout training and testing; the simulations draw p_k once and use those exact values as the oracle reliability. The paper does not test sensitivity to forecast error, drift over time, or strategic misreporting. Since the entire feature/embedding allocation is ordered by these forecasts, incorrect forecasts could reorder assignments and reverse the gains. Add experiments with perturbed or time-varying p_k, or clearly state that the claimed benefits hold only under perfectly accurate, static reliability forecasts.","section":"Section V-A / VI-B (reliability forecasts)"},{"comment":"All conclusions rest on averages over 10 simulation runs with no standard errors, confidence intervals, or significance tests, and the comparison uses a single baseline. This is insufficient support for the headline 14.5%/18% effect, particularly in light of the checkpoint-selection issue. Please report per-run distributions, error bars, and a paired test over the 10 runs, and consider at least one additional baseline (e.g., equal embedding dimensions with reliability-aware feature assignment, or random feature assignment with matched total embedding budget).","section":"Section VI.B-C (statistical reporting)"}],"minor_comments":[{"comment":"The sentence claiming the baseline 'represents an upper bound on the performance of the algorithm described in [10]' is not justified, since the experimental instantiation of [10] is not described (e.g., whether dropout-adaptation training is used). Clarify the baseline configuration.","section":"Section VI"},{"comment":"The feature-importance and embedding-scaling rule is stated qualitatively ('in proportion to their reliability', 'scale linearly with the client’s reliability'); specify the exact mapping used in the experiments for reproducibility.","section":"Section V-B"},{"comment":"Typos and wording: 'alligns' (Section IV), 'its'' (Sections IV and V), 'via using' (Section VI-B), and inconsistent capitalization of 'NWDAFs'/'NWDAF server' should be corrected.","section":"Sections IV, V, VI"},{"comment":"The notation a_k is used both for the Bernoulli availability indicator and in the ID definition without an explicit definition of its range; define it once in Section VI-B.","section":"Section VI-B"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The test-set checkpoint selection and the absolute-value percentage are serious but fixable evaluation issues; I would not reject on them alone. The paper is a modest empirical contribution and may fit a venue that values applied systems work. Please ensure the revision includes a validation-split selection protocol, signed effect sizes with variability, and sensitivity to reliability forecast error, since the current version substantially overstates confidence in the reported gains. The single-baseline comparison and lack of significance testing should also be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the mechanism is genuinely new in this setting: the server centrally assigns features by importance and scales each client's local model and embedding size by the client's reliability forecast, exploiting centralized data management in 5G core networks. It is a sensible engineering heuristic and a reasonable step beyond the dropout-simulation and serverless baselines in [10] and [9]. Second, the reported 14.5% and 18% loss reductions are not yet believable as expected deployed gains, because the checkpoints are chosen using test-set performance (Section VI.B) and no error bars or significance tests are given. The stress-test note lands: this is a real evaluation leak.\n\nThe paper does several things well. The system model is clearly written, the algorithm is simple to implement, and the code and dataset are released. The authors are also honest that gains concentrate in availability patterns where reliable clients participate (Fig. 4, ID >= 8). For a practitioner in 5G core networks, the idea is plausible and cheap to try.\n\nThe soft spots, in proportion. First and main: Section VI.B says the server continually stores models with the best test performance during training and reverts to those top performers. That makes the headline numbers best-of-run over the test set, not the expected performance of a deployed model. The same protocol is used for the baseline, so it is not a relative unfairness, but it is a biased estimate of absolute gain, and if the proposed method's per-round trajectory is noisier—plausible, since capacity is concentrated in a few clients—best-of-run selection can systematically favor it. The fix is straightforward: use a validation split for checkpoint selection and evaluate once on test. The 10-run averages should be accompanied by standard errors or confidence intervals; per-ID differences already flip sign (e.g., Fig. 4 at IDs 2–6), so the average gain needs an interval.\n\nSecond, the method assumes each NWDAF can forecast its reliability for the whole training period and that these probabilities stay fixed (Sections V-A and VI-B). The paper does not test sensitivity to forecast error or drift. This is a separate load-bearing premise: if reliability tags are misordered, the allocation could underperform a random split. A simple perturbation experiment would address it.\n\nMinor: one baseline and one dataset (proprietary, though the GitHub release mitigates this), and feature importance is computed with a basic decision tree before training; no analysis of that choice.\n\nBottom line: a plausible engineering contribution, clearly written and reproducible, but the quantitative claim needs re-evaluation with a validation split and uncertainty quantification. The paper is for researchers working on VFL robustness in 5G core networks, or anyone using feature allocation under reliability constraints. I would send it to review: the mechanism is well defined, the flaw is fixable, and the contribution is non-trivial. But I would not take the 14.5%/18% numbers at face value until the evaluation is corrected.\n\nRecommendation: major revision requiring a held-out validation split, confidence intervals, a sensitivity analysis on reliability forecast error, and ideally an extra baseline.","headline":"A sensible reliability-aware feature allocation heuristic for VFL in 5G core networks, undermined by test-set checkpoint selection and missing uncertainty quantification; fixable with a validation split.","tokens_in":9883,"tokens_out":3736,"would_cite":false,"duration_ms":32571,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that reliability-aware feature allocation reduces vertical federated learning loss in 5G core networks by assigning important features and larger embedding dimensions to more dependable clients.","keywords":["vertical federated learning","client reliability","feature importance","feature association","model definition","5G core network","NWDAF","generalization loss"],"falsifier":"Re-run the same simulations with reliability forecasts corrupted by noise or drifting mid-training; if the average weighted test loss improvement over the baseline shrinks or reverses once forecast error passes a small threshold, the central claim collapses. A cleaner separation test is to compare against a baseline that keeps the same per-client embedding dimensions but assigns features randomly, which would isolate whether the gain comes from feature-importance matching or embedding sizing.","tokens_in":8906,"feed_emoji":"📡","tokens_out":8751,"duration_ms":70240,"temperature":0.7,"pith_summary":"Vertical federated learning (VFL) lets several data holders train a shared model on different feature sets, but in a 5G core network the analytics functions acting as clients can drop out under load, and the missing embeddings degrade the model. This paper proposes that the central server should assign features and local-model sizes by client reliability: more important features and larger embedding dimensions go to clients with higher forecast reliability. In simulations of a video-quality prediction task with four clients, success probabilities drawn from Beta(8,2) (mean 0.8) and Beta(10,6) (mean 0.625), the proposed method reduces average weighted test loss by about 14.5% and 18% respectively relative to a dropout-simulation baseline. If the result holds, reliability-aware feature allocation is a low-cost way to make VFL robust in centralized network environments, without changing the federated training loop.","feed_headline":"Reliability-aware feature split cuts vertical federated loss 18%","feed_subtitle":"Assigning important features to dependable analytics functions keeps predictions accurate when overloaded clients drop out.","key_machinery":"The mechanism is a feature association and model definition algorithm executed before training. The server first has an analytics function compute feature-importance scores for the centrally stored data, then assigns features to each client so that total assigned feature importance and embedding dimension both scale linearly with that client's reliability probability, subject to a fixed total embedding budget. The server model's input dimension is set by the concatenation of these per-client embedding sizes, and dropouts in the training loop cause unavailable clients' embeddings to be replaced by zero vectors. This couples the information capacity of each client's contribution to the probability that the contribution arrives.","core_discovery":"In a VFL system with clients of unequal reliability, the way features are partitioned across clients is itself a source of robustness. The paper's central claim is that a server with centralized control over a feature pool can mitigate dropout-induced generalization loss by assigning each client a local model whose feature set and embedding dimension are scaled to that client's forecast reliability. In the evaluated 5G core-network scenario, this reliability-aware assignment yields an average weighted test loss about 14.5% lower than the dropout-simulation baseline under Beta(8,2) reliability and about 18% lower under Beta(10,6) reliability. The improvement concentrates in availability patterns where the most reliable client, carrying the most important features, participates in inference.","pith_inferences":["The paper does not vary forecast error, so a fair next experiment would corrupt the reported reliability measures and map how much misordering the algorithm tolerates before it falls behind the baseline.","A natural extension is to re-estimate reliability online and periodically reassign features or adjust embedding sizes, which the static initialization here does not allow.","If reliability forecasts are self-reported, clients have an incentive to overstate reliability to receive important features; an incentive-compatible or verified-reporting mechanism would be needed in adversarial deployments.","The same reliability-proportional capacity allocation could apply within a single client's model, for example by giving more network capacity to features that come from dependable data sources."],"forward_implications":["Operators can adopt the initialization phase as a lightweight add-on to existing NWDAF-based VFL, since the training and inference loops are unchanged.","The method is most valuable when client reliability is moderate and varied; in the high-reliability scenario the gain is smaller but still about 14.5%.","A fixed total embedding budget can be redistributed toward reliable clients without extra communication cost, effectively buying robustness without spending more bandwidth.","The gains depend on centralized data orchestration, so the algorithm transfers only to environments where a server can reassign features among clients."],"supporting_citations":[{"why":"Supplies the baseline VFL method that randomly distributes features and simulates dropout, against which the reliability-aware method is measured.","marker":"[10]"},{"why":"Provides the Decision Tree Regressor used in the experiments to compute feature importances for the proposed feature-assignment step.","marker":"[17]"},{"why":"Provides the federated-learning simulation framework used to implement the training and comparison.","marker":"[16]"},{"why":"Defines the centralized and distributed NWDAF architecture that motivates the server's ability to manage a feature pool centrally.","marker":"[11]"},{"why":"Represents prior VFL reliability work that the paper contrasts with, because it assumes fully-observed training data and no central feature assignment.","marker":"[9]"},{"why":"Gives the vertical federated learning formulation and training objective that the proposed algorithm operates within.","marker":"[2]"}],"fun_headline_variants":["Reliability-based feature split lifts VFL accuracy in 5G cores","Feature assignment by reliability cuts VFL loss up to 18%","5G core VFL: reliability-aware split trims test loss","In VFL, assigning features by client reliability reduces loss","Dropout-proof VFL: reliability-driven feature split in 5G CN"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each client can accurately forecast its reliability for the whole training period and that these probabilities stay fixed during training and testing, because noisy or drifting forecasts would misorder feature assignment and could reverse the reported gains.","fun_headline_variants_meta":{"raw":{"variants":["Reliability-based feature split lifts VFL accuracy in 5G cores","Feature assignment by reliability cuts VFL loss up to 18%","5G core VFL: reliability-aware split trims test loss","In VFL, assigning features by client reliability reduces loss","Dropout-proof VFL: reliability-driven feature split in 5G CN"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1270,"prompt_tokens":849,"completion_tokens":421,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":465,"completion_tokens_details":{"reasoning_tokens":339}},"tokens_in":465,"tokens_out":421,"duration_ms":4069,"temperature":1.0,"reasoning_tokens":339,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:20:34.212845+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same simulations with reliability forecasts corrupted by noise or drifting mid-training; if the average weighted test loss improvement over the baseline shrinks or reverses once forecast error passes a small threshold, the central claim collapses. A cleaner separation test is to compare against a baseline that keeps the same per-client embedding dimensions but assigns features randomly, which would isolate whether the gain comes from feature-importance matching or embedding sizing.","supporting_citations":[{"cited_title":"Robust and IP-Protecting Vertical Federated Learning against Unexpected Quitting of Parties","cited_arxiv_id":"2303.18178","evidence_quote":"Supplies the baseline VFL method that randomly distributes features and simulates dropout, against which the reliability-aware method is measured."},{"cited_title":"Pedregosa et al., ”scikit-learn: Machine Learning in Python,” Journal of Machine Learning Research, vol","cited_arxiv_id":null,"evidence_quote":"Provides the Decision Tree Regressor used in the experiments to compute feature importances for the proposed feature-assignment step."}],"review_version":1}