{"id":"9a1ba6a4-db14-43b3-87c3-355957a7b821","arxiv_id":"2412.08950","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A federated-learning FPS distribution predictor using per-player and per-game learnable kernels achieves a Wasserstein distance of 0.469, but the evaluation may leak identity information.","lead":"This paper combines a global telemetry dataset of 100,000 players with a federated learning model to predict how many frames per second a game will run on a given player's machine. It reports better accuracy than existing checklists and binary tools, but the privacy and cold-start claims need stronger evaluation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Random 80/20 split leaks player and game identity through per-entity LKKs, so the headline 0.469 and 7.57% kernel gain do not demonstrate cold-start performance.","rationale":"The reader's weakest assumption identifies exactly the load-bearing concern: the random 80/20 split of player-game pairs, combined with per-entity LKKs, means validation entities overlap with training entities. The paper's strongest claims are the absolute predictive performance and the kernel-induced improvement, both of which depend on this evaluation. In the ablation, the 'w/ both' condition uses both player and game kernels; under the random split, those kernels have seen training data for the same player and game, so the 7.57% reduction relative to 'w/o' does not establish generalization to unseen players or games. The cold-start discussion in Sections 5.2 and 7.2 makes this even more central: the deployment scenario is precisely new entities with few or no records, but the validation set does not simulate that scenario. A grouped split by player and game is a straightforward, decisive check. If the improvement persists under grouped splits and only absolute performance drops modestly, the core contribution survives with appropriately scoped claims. If the improvement disappears, the kernel contribution is memorization rather than generalization. I find no other concern as load-bearing: the lack of error bars, closed data, and ambiguous federated aggregation of game LKKs are secondary and would be partially resolved by the same re-evaluation. I therefore agree with the reader's conditional verdict and recommend the same conditional acceptance with the grouped-split re-analysis as a required revision.","tokens_in":18836,"tokens_out":2128,"duration_ms":24600,"concrete_test":"Re-run the full federated training pipeline using grouped splits: (1) hold out all player-game pairs for 20% of players; (2) hold out all pairs for 20% of games; (3) hold out all pairs for 20% of players and 20% of games jointly. For each split, report Wasserstein distance, cross-entropy, and the w/ both vs w/o LKK ablation, with mean and standard error over at least five random seeds. If the 0.469 WD and 7.57% kernel gain degrade substantially under any grouped split, the claims should be reframed as in-dataset interpolation rather than cold-start prediction.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 6.3 states the dataset was 'randomly split into 80% training and 20% validation subsets,' while Section 5.2 assigns a unique learnable knowledge kernel (LKK) to each player and each game. Under a random split of player-game pairs, the same player and the same game can appear in both training and validation. For a validation pair, the player-specific LKK is trained on that player's other game sessions, and the game-specific LKK is trained on other players' sessions for the same game. The model can therefore look up per-entity FPS tendencies rather than predict them from features. This directly inflates the headline mean Wasserstein distance of 0.469 (Table 5) and, more importantly, the 7.57% improvement attributed to LKKs (Table 6: w/ both 0.4690 vs w/o 0.5074). The paper's stated goal is to address cold-start for new players and games; the random split does not measure cold-start because validation entities are not new. The 7.57% gain may partly measure memorization of entity identity. The claimed plug-and-play cold-start scheme is therefore not validated by the reported experiments. This is a correctness risk, not merely a robustness footnote, because the central contributions are the per-entity kernels and cold-start handling.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper tackles the problem of predicting, before purchase, the FPS distribution a player will experience in a given game. The authors contribute a large telemetry dataset (76.4 million game processes from 100,000 users across 224 countries and regions, covering 835 games), an analysis of micro-level (device type, CPU/GPU, OS) and macro-level (GDP per capita, Gini index) determinants of the 95% FPS floor, and a federated-learning MLP predictor in which every player and every game is assigned a trainable learnable knowledge kernel (LKK). The model is trained with a four-branch loss that averages predictions with and without each kernel, and a plug-and-play kernel-switching scheme is proposed to address cold start. Reported results include a mean Wasserstein distance of 0.469 (Table 5), a 7.57% improvement from using both kernels (Table 6), and an auxiliary 42-class FPS-distribution predictor. The paper's central evaluation claim is that the per-entity kernels and the federated strategy improve or match centralized baselines while handling cold start, and that the resulting model outperforms standard machine-learning baselines.","tokens_in":19122,"tokens_out":12244,"duration_ms":107205,"significance":"The dataset and the statistical analysis are the paper's clearest contributions: the scale (100K users, 835 games, 76.4M sessions) exceeds prior public work in this space, and the ANOVA/regression results on GPU/CPU/device and country-level socio-economic factors give practitioners concrete variable-selection guidance. The proposed architecture is lightweight (1.44 MB, sub-millisecond inference) and the comparison between centralized and federated training is a reasonable design choice. If the evaluation is repaired, the paper would provide a credible benchmark for device-aware FPS prediction. However, the headline numeric claims are currently conditional on the split-leakage fix: as reported, the 0.469 Wasserstein distance and the 7.57% kernel gain do not establish cold-start capability, and no uncertainty estimates support the small differences the paper emphasizes. I also note that the dataset and model code are not public, which limits independent verification; this is a limitation, not a defect.","major_comments":[{"comment":"The random 80/20 split of player-game pairs described in §6.3 is incompatible with the paper's cold-start claims, and the stress-test concern lands. Because §5.2 attaches a unique trainable LKK to every player and every game, a validation pair can involve a player whose LKK was trained on that player's other sessions in the training set and a game whose LKK was trained on other players' sessions of the same game. The validation entities are therefore not new, so the mean Wasserstein distance of 0.469 (Table 5) and the 7.57% gain attributed to the kernels (Table 6: 0.5074 without kernels vs 0.4690 with both) do not measure cold-start performance; the gain may partly reflect per-entity memorization of FPS tendencies. Moreover, the baseline methods in Table 5 have no entity-specific parameters, so the comparison is not apples-to-apples for the cold-start setting. The evaluation should be redone with entity-disjoint splits (held-out players and held-out games evaluated separately, including the zero-record case), and the kernel-vs-no-kernel comparison should be reported within those splits.","section":"§6.3, §5.2"},{"comment":"All metrics are reported as single numbers with no error bars, confidence intervals, or significance tests. The differences that support the paper's claims are small: centralized 0.4698 vs federated 0.4690 in Table 5, and w/o kernels 0.5074 vs w/ game kernel only 0.4984 in Table 6. Without multiple seeds or paired significance testing, the 7.57% improvement and the conclusion that federated training does not degrade performance cannot be distinguished from run-to-run noise.","section":"§6.1–§6.3, Tables 5–6"},{"comment":"The plug-and-play cold-start protocol described in §5.2 (requiring at least three trained records for a player and ten records for a game before enabling the corresponding kernel) is never exercised in the reported experiments. Table 6 ablates fully trained kernels on the validation set, which tests enrichment by trained kernels rather than the cold-start regime for new players or games with few or zero prior records. The paper should evaluate the threshold-based protocol as a function of the number of available records per player and per game, including the strictly zero-record case.","section":"§5.2, §6.2"},{"comment":"The federated setup is under-specified, which makes the centralized-vs-federated parity result in Table 5 non-reproducible. The paper does not state how the 100,000 users map to FL clients, the number of communication rounds, the client sampling and aggregation strategy, or how game-level LKKs are updated from the distributed player devices even though a game kernel is shared across the players of that game; the statement that kernels are 'distributed and trained on each player's device' is ambiguous for shared game kernels.","section":"§5.2, §6.3"}],"minor_comments":[{"comment":"Typos and inconsistent labels should be cleaned up: 'influnced' and 'and and statistics-based conclusion' (§2.1), 'centeralized/fedrated' (Table 5 caption), 'Fedrate' (Fig. 1), 'Auxillary' (§6.4), 'optimazation' (§7.3), and the malformed term 'log10α+− 0.42β' in Eq. (2).","section":"§2.1, §7.3, Table 5, Fig. 1, Eq. (2)"},{"comment":"The loss subscripts L_wg, L_wp, L_wb, L_wo in Fig. 7(b) are never defined; the text should state that these denote the losses with the game kernel, with the player kernel, with both kernels, and without kernels, respectively.","section":"§5.2, Fig. 7"},{"comment":"p-values are reported as 0.000 or p = 0; these should be expressed as p < 0.001, and the ANOVA in Table 3 should clarify how games with multiple tags per categorical feature were entered into the analysis, since the note that 'the degree of freedom is not static' makes the design ambiguous.","section":"§4, Tables 1–4"},{"comment":"The data-exclusion cutoffs (sessions shorter than 5 minutes, players with fewer than 18 game processes, games with fewer than 10 records) are presented without justification; a brief rationale or sensitivity analysis would strengthen confidence in the preprocessing.","section":"§3"},{"comment":"The text reports the confusion matrix for the 'best centralized and federated trained models' without defining the selection criterion (for example, best validation Wasserstein distance across epochs); the criterion should be stated.","section":"§6.1"},{"comment":"The claim that sharing only gradients 'ensures user privacy' overstates what plain federated averaging provides; without differential privacy or an explicit threat model, the paper should say that the design reduces data exposure rather than guarantees privacy.","section":"§2.2, §5.2, §8"}],"recommendation":"major_revision","confidential_remarks":"Two editorial points. First, the dataset is not public and the authors even withhold their institution's name until publication (Section 8), so the empirical claims rest on data that reviewers and readers cannot inspect; independent verification of the headline numbers is impossible, and I would ask the editor to weigh this against the journal's reproducibility expectations. Second, the paper's framing as a cs.HC contribution is loose: the QoE link is imported from prior work [38], and the paper's own contribution is largely a data-collection, statistical-analysis, and ML-systems effort; acceptance should be judged on those terms. I found no citation-pattern concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: a genuinely useful dataset in search of a trustworthy evaluation. The paper collects 76.4M game-process records from 100k users across 835 games, which is a real contribution to the game QoE literature. The descriptive analysis of FPS determinants (hardware, country GDP, Gini) is reasonable and mostly well done. The federated learning framing is sensible, and the per-entity kernels are an acceptable way to capture unmeasured player/game heterogeneity, even if they are technically entity embeddings with a clever training scheme.\n\nThe soft spot is the one the stress test flags, and it's load-bearing. Section 6.3 reports an 80/20 random split of player-game pairs, but Section 5.2 gives each player and each game its own learnable kernel. Under that split, the same player and the same game appear in both training and validation. For a validation pair, the player kernel was trained on that player's other sessions, and the game kernel on other players' sessions for the same game. So the reported 0.469 Wasserstein and the 7.57% gain from using both kernels are partly memorization of entity identity, not prediction for new players or games. The paper's stated cold-start goal is not measured by this protocol. The fix is a grouped split (e.g., hold out players and games), plus error bars or significance tests around the table metrics. Without that, the headline numbers should not be taken at face value.\n\nTwo smaller points. The data is closed and the code isn't shipped, so the numbers can't be independently checked; understandable given the confidentiality, but it does raise the bar for internal consistency. And the privacy story is under-specified: if the game LKK is shared across players, how is it aggregated in the federated scheme? The paper doesn't say, which matters because that's the part that could leak information about individual sessions.\n\nBottom line: this is a solid, interesting paper with a flawed central evaluation. The dataset and task formulation are worth publishing, and a serious referee should see it. I'd send it to review, but with a clear request to re-run the experiments with a grouped split and report uncertainty. My own verdict: conditional, not accept.","headline":"Valuable telemetry dataset and a plausible task, but the random split leaks entity identity through per-player/game kernels, so the headline numbers don't support the cold-start claim.","tokens_in":19666,"tokens_out":2386,"would_cite":true,"duration_ms":23184,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a privacy-preserving federated model can predict the FPS distribution a player will experience for a given game, achieving a mean Wasserstein distance of 0.469 against ground truth and a 7.57% error reduction from…","keywords":["frames per second","gaming experience","federated learning","telemetry data","FPS distribution prediction","learnable knowledge kernel","cold start","Wasserstein distance"],"falsifier":"A strict split by player identity and game identity, with no overlap between training and validation entities, would test the generalization claim; if the Wasserstein distance rises toward the no-kernel baseline or the 7.57% kernel gain disappears, then the reported cold-start capability is an artifact of entity leakage.","tokens_in":18619,"feed_emoji":"🎮","tokens_out":8561,"duration_ms":84736,"temperature":0.7,"pith_summary":"The paper sets out to show that a game's frame rate performance on a player's machine can be predicted as a distribution, not just as a pass/fail or a spec list, and that this can be done without moving raw player telemetry to a central server. A reader would care because pre-purchase FPS estimates could prevent wasted purchases and negative reviews, while the federated design keeps sensitive hardware, software, and session data on the player's device. The authors support this with a 76.4-million-session dataset from 100,000 players in 224 countries and 835 games, then train a model whose per-player and per-game knowledge kernels capture unquantifiable factors. Headline results are a mean Wasserstein distance of 0.469 and a 7.57% kernel-driven error reduction.","feed_headline":"Federated model predicts game FPS without collecting raw player data","feed_subtitle":"Learned per-player and per-game profiles cut prediction error by 7.57% while keeping data on devices.","key_machinery":"The central object is the learnable knowledge kernel (LKK), a compact per-player and per-game embedding vector trained locally and aggregated by federated averaging. At each training step the network computes four outputs: no kernel, player kernel only, game kernel only, and both kernels; merge layers concatenate each kernel to the feature map and add the fused result through a skip connection, so the kernel refines rather than replaces the features. The four losses are averaged, which forces the network to remain accurate even when an identity's kernel is absent, and at inference a rule (e.g., at least three player records or ten game records) decides whether to plug a kernel in. The evaluation's primary metric is Wasserstein distance, which measures how much probability mass must be shifted to turn the predicted FPS distribution into the ground truth.","core_discovery":"The paper claims that FPS prediction should be treated as a distribution-prediction problem at the player-game level, and that a federated neural network with identity-specific embeddings is the first method to solve it at global scale. It reports that a centralized version achieves Wasserstein distance 0.4698 and the federated version 0.4690, with cross entropy 1.3871, outperforming softmax regression, decision trees, random forest, and XGBoost. It further claims that the learnable knowledge kernels are not decorative: ablating both kernels raises Wasserstein distance to 0.5074, while using both brings it to 0.4690, a 7.57% improvement, and that the dynamic kernel switch preserves this benefit for cold-start users and games. It also establishes that macro-level economic indicators, GDP per capita (log) and Gini index, explain 62.5% of country-level variance in the 95% FPS floor, yielding a simple formula.","pith_inferences":["Beyond the paper: the random 80/20 split means the reported 0.469 is partly an interpolation score; a strict player-disjoint and game-disjoint split would likely produce a higher distance and is the right benchmark for real storefront use.","The ablation results suggest the player kernel contributes more than the game kernel (0.4781 vs 0.4984 in Wasserstein distance), so a platform serving only one kernel type could reasonably prioritize player-history features.","A natural testable extension is to reuse the same LKK machinery for other experience outcomes such as stutter frequency, session length, or input latency, and to warm-start a newly released game from kernels of similar-genre titles."],"forward_implications":["If the 0.469 Wasserstein result holds in deployment, game storefronts could show a personalized, pre-purchase FPS distribution instead of a static minimum/recommended spec list.","Because federated training matches centralized accuracy on the reported headline metrics, a deployment can keep raw telemetry on devices and share only gradients, making the service compatible with stricter privacy rules.","The 42-bin version extends the same architecture to fine-grained distributions, supporting stutter-frequency and stability analyses for players who want more than a coarse rating.","The dynamic kernel switch means a new player or game can receive a cold-start prediction immediately, with the prediction sharpening as a few game records accumulate.","The macro-level correlations imply that regional rollouts could use GDP per capita and Gini index as a prior before device-level data arrives."],"supporting_citations":[{"why":"Establishes the 95% FPS floor as a better quality-of-experience predictor than mean FPS, defining the most important region of the FPS distribution.","marker":"[38]"},{"why":"Shows that frame rate changes affect player performance in first-person shooters, grounding FPS as the central experience metric.","marker":"[22]"},{"why":"Documents how frame rate and resolution shape game experience, informing the micro-level player-side and game-side feature set.","marker":"[23]"},{"why":"Provides the XGBoost baseline that the proposed model must outperform in the main comparison table.","marker":"[21]"},{"why":"Supplies the federated-learning foundation for training across decentralized devices without sharing raw data.","marker":"[35]"},{"why":"Defines the cold-start problem in recommender-like systems, the problem the kernel switch is designed to solve.","marker":"[36]"},{"why":"Supplies the GPU performance classification used to construct the GPU-level feature in the analysis.","marker":"[28]"},{"why":"Provides the one-way ANOVA method used to test which device and game features significantly affect the 95% FPS floor.","marker":"[47]"}],"fun_headline_variants":["Federated learning predicts FPS with 0.469 Wasserstein distance","Privacy-preserving FPS forecasts beat baselines by 7.57%","Global telemetry + federated learning: FPS prediction without raw data","FPS prediction goes federated: 7.57% better, zero data leaks","Cold-start solved: dynamic kernels for global FPS prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes an 80/20 random split of player-game pairs tells us how well the model generalizes to players and games it has never seen, but the same player and the same game can appear on both sides of the split, so the validation is not a clean cold-start test.","fun_headline_variants_meta":{"raw":{"variants":["Federated learning predicts FPS with 0.469 Wasserstein distance","Privacy-preserving FPS forecasts beat baselines by 7.57%","Global telemetry + federated learning: FPS prediction without raw data","FPS prediction goes federated: 7.57% better, zero data leaks","Cold-start solved: dynamic kernels for global FPS prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000709,"raw_usage":{"total_tokens":3204,"prompt_tokens":969,"completion_tokens":2235,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":2137}},"tokens_in":585,"tokens_out":2235,"duration_ms":16984,"temperature":1.0,"reasoning_tokens":2137,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:22:19.115431+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A strict split by player identity and game identity, with no overlap between training and validation entities, would test the generalization claim; if the Wasserstein distance rises toward the no-kernel baseline or the 7.57% kernel gain disappears, then the reported cold-start capability is an artifact of entity leakage.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the 95% FPS floor as a better quality-of-experience predictor than mean FPS, defining the most important region of the FPS distribution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that frame rate changes affect player performance in first-person shooters, grounding FPS as the central experience metric."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents how frame rate and resolution shape game experience, informing the micro-level player-side and game-side feature set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the federated-learning foundation for training across decentralized devices without sharing raw data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GPU performance classification used to construct the GPU-level feature in the analysis."}],"review_version":1}