{"id":"3932d8f8-285d-4547-990b-c2f5632cca52","arxiv_id":"2412.19069","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A thesis shows that a pairwise gradient-based online ranker can be federated as FPDGD, and analyzes non-IID data, poisoning attacks, and unlearning in federated online learning to rank.","lead":"This PhD thesis proposes FPDGD, a federated version of an online learning to rank algorithm that learns from clicks while keeping user data on devices, and studies non-IID data, poisoning attacks, and client removal in such systems. It reports that FPDGD outperforms the previous federated method on large datasets, while certain non-IID patterns hurt performance and approximate unlearning can reduce a leaving client's influence.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FPDGD's ε-DP claim uses L2 norm clipping with Laplace noise, but Laplace sensitivity is L1; for a d-dimensional model the effective sensitivity is Δ√d, so the stated privacy budget is not achieved.","rationale":"The reader's weakest assumption correctly identified the differential privacy guarantee as the most fragile part of the central claim. My analysis agrees that the ε-DP assertion is not established, but the precise reason is sharper: the clipping operation in Eq. 3.19 bounds L2 norm, whereas the Laplace mechanism's sensitivity is L1. Thus even if Δ were derived from a full sensitivity analysis of PDGD, the stated noise scale would not deliver the claimed ε. This concern is load-bearing because the thesis's headline contribution is an 'effective and privacy-preserving' FOLTR method, and the 'secured' wording is part of the strongest claim. The effectiveness comparison against FOLtR-ES is supported by extensive simulated experiments and published peer-reviewed work, so I would not reject or down-rank the thesis on that basis. The correct remedy is to require a corrected privacy analysis or an explicit statement of the achieved (weaker) privacy bound before the security claim is taken as established. The reader's CONDITIONAL verdict remains appropriate; my stress-test does not move it.","tokens_in":51368,"tokens_out":5549,"duration_ms":59683,"concrete_test":"Compute the actual L1 sensitivity of the L2-clipped FPDGD aggregate empirically: train FPDGD with Eq. 3.19 on pairs of datasets that differ in exactly one client's local data, and record the maximum L1 norm of the difference between the released global models. If this maximum exceeds Δ (or Δ/n when reporting the average model), the noise scale Δ/ε yields an effective ε larger than claimed. An analytical cross-check: for a d-dimensional model, the maximum L1 difference between two vectors with L2 norm ≤ Δ/2 is Δ√d; for MSLR-WEB10K's 136-dimensional linear ranker this already gives effective ε ≈ 14.0 at claimed ε = 1.2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central 'secured' claim rests on the ε-DP guarantee in Section 3.2.1. The mechanism clips each client's local weight vector θ to L2 norm Δ/2 (Eq. 3.19) and adds Gamma-split Laplace noise with scale Δ/ε. However, the Laplace mechanism requires the global sensitivity in L1 norm, not L2. Two vectors each with L2 norm at most Δ/2 can differ in L1 norm by up to Δ√d, where d is the model dimension. Therefore the L1 sensitivity of the aggregate global model is at least Δ√d (or Δ√d/n for the average), not Δ. Consequently, adding noise with scale Δ/ε provides only ε√d-DP, not ε-DP. For the linear ranker on MSLR-WEB10K, d = 136, so a claimed ε = 1.2 corresponds to an effective privacy budget of roughly 14.0; for neural rankers the gap is larger. The fact that Δ is chosen by grid search is not itself the flaw, since clipping does bound sensitivity in L2, but the norm mismatch between the clipping bound and the Laplace sensitivity means the stated privacy guarantee does not follow from the presented analysis. This is a load-bearing concern because the thesis explicitly claims FPDGD is a 'secured' and 'privacy-preserving' alternative, not merely an effective one.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The thesis studies Federated Online Learning to Rank (FOLTR), motivated by privacy concerns in centralized OLTR. It makes four contributions: (1) an analysis of the existing FOLtR-ES method, showing instability on large datasets; (2) a new method FPDGD, which adapts PDGD to the FedAvg framework and adds a differential privacy mechanism via norm clipping and Gamma-split Laplace noise; (3) an empirical taxonomy and study of non-IID data in FOLTR; (4) a study of poisoning attacks and defenses, and an unlearning method with a poisoning-based verification. The effectiveness comparison of FPDGD against FOLtR-ES is supported by extensive experiments on MQ2007 and MSLR-WEB10K with significance tests, and the thesis honestly reports settings where FOLtR-ES wins (Table 3.1, MQ2007 perfect and navigational clicks). However, the central 'secured' claim rests on a differentially private mechanism whose stated privacy guarantee does not follow from the presented analysis, because the sensitivity is computed in the L2 norm while Laplace noise requires L1 sensitivity.","tokens_in":51693,"tokens_out":3845,"duration_ms":36442,"significance":"If the privacy guarantee were valid, FPDGD would be the first gradient-based FOLTR method with consistent gains over the evolutionary-strategy baseline, and the thesis would make a solid contribution to privacy-preserving ranking. The non-IID taxonomy (Type 1--4), the poisoning attack/defense analysis, and the unlearning verification approach are useful empirical benchmarks for a nascent subfield. The thesis is also commendable for releasing code and experimental scripts for most chapters, and for including statistical significance testing and honest reporting of negative results (e.g., MQ2007 cases where FOLtR-ES is better). However, the privacy analysis in Chapter 3 is load-bearing for the thesis's claim that FPDGD is 'secured' and 'privacy-preserving', and that analysis is incorrect as presented; this materially weakens the contribution.","major_comments":[{"comment":"The differential privacy mechanism clips each local weight vector to L2 norm Delta/2 and adds Gamma-split Laplace noise with scale Delta/epsilon. The Laplace mechanism (Definition 3.2.2) requires the global sensitivity in the L1 norm, not the L2 norm. Two d-dimensional vectors each with L2 norm at most Delta/2 can differ in L1 norm by up to Delta*sqrt(d). For the linear ranker on MSLR-WEB10K, d = 136, so a claimed epsilon = 1.2 corresponds to an effective privacy budget of at least 1.2*sqrt(136) ≈ 14.0; for the neural ranker the gap is larger. The statement in Section 3.2.1 that clipping to Delta/2 'can meet the global sensitivity Delta' is therefore unsupported. The claimed epsilon-DP guarantee does not follow from the presented analysis, undermining the central 'secured' and 'privacy-preserving' claims of the thesis. The mechanism should either clip in L1 norm, calibrate the noise to the true L1 sensitivity Delta*sqrt(d), or use a different privacy accounting such as the moments accountant with L2 sensitivity and Gaussian noise.","section":"Section 3.2.4 and Table 3.1"},{"comment":"The summary states that 'Empirical evaluation shows FPDGD significantly outperforms the only other federated OLTR method (i.e., FOLtR-ES)'. Table 3.1 shows the opposite for MQ2007 under the perfect and navigational click models, where FOLtR-ES is significantly better than FPDGD (marked with ▼, p < 0.01) for all privacy levels epsilon. The claim should be qualified to the large-scale dataset (MSLR-WEB10K) or to the informational click model on MQ2007. As written, the claim is contradicted by the thesis's own results and overstates the reliability of FPDGD.","section":"Section 3.2.4 and Table 3.1"},{"comment":"The unlearning verification injects a poisoning attack designed by the authors (a reversed CCM click model and amplified updates) and measures whether its impact diminishes after unlearning. This is a plausible approach, but it only verifies removal of a contrived signal, not the removal of a client's actual data contributions. The thesis should explicitly state that the verification is conditional on the attack model and that it does not provide a formal or general guarantee of unlearning. This limitation is important because Chapter 6 claims, based on this verification, that the proposed method effectively forgets client contributions.","section":"Section 6.1.4 and Section 6.3.1"}],"minor_comments":[{"comment":"Typo: 'datesets' should be 'datasets'.","section":"Definition 3.2.1"},{"comment":"Typo: 'outputed' should be 'output'.","section":"Section 3.2.1"},{"comment":"The mapping from FOLtR-ES privacy parameter p in {0.25, 0.5, 0.9, 1.0} to epsilon values in {1.2, 2.3, 4.5, 10} is asserted but not derived. Please show the calculation using Eq. (3.6) and state the number of discrete metric values n used in that formula.","section":"Section 3.2.2"},{"comment":"The conclusions that Type 3 (click preferences) and Type 4 (data quantity) non-IID data do not impact FPDGD effectiveness are based on experiments with a linear ranker and the MSLR-WEB10K dataset only. Please qualify the generality of these findings, as the thesis itself notes for Type 4 that the result 'may be specific to FPDGD'.","section":"Section 4.5-4.6"}],"recommendation":"major_revision","confidential_remarks":"The thesis is a compilation of previously published papers, and the DP sensitivity issue appears to stem from the underlying ICTIR 2021 paper. The effectiveness comparison and the empirical analyses are solid and well-documented, but the privacy guarantee is the central advertised contribution of Chapter 3 and is not valid as stated. If the authors can correct the DP analysis (e.g., by using L1 clipping or recalibrating the noise to the actual L1 sensitivity), the thesis would be much stronger. The overstatement in Section 3.2.4 should also be corrected. These issues are fixable, so major revision rather than reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the one thing you should know: this is a PhD thesis that compiles five peer-reviewed papers by the author. There are no new results beyond those papers, so if you need the latest, go to the cited publications. But as a single document that maps the FOLTR landscape—effectiveness, non-IID, poisoning, unlearning—it is genuinely useful.\n\nThe strongest part is the empirical work on FPDGD. The comparisons against FOLtR-ES are careful, use standard OLTR datasets and click models, run multiple repeats with significance tests. The thesis honestly reports cases where FOLtR-ES wins, such as MQ2007 navigational clicks. The four-type non-IID taxonomy is a real contribution, even though only Types 1 and 2 (partially) hurt performance. The poisoning study is a solid map of attack/defense interactions. The unlearning method is a first for FOLTR and the efficiency argument is plausible. Code links for the experiments are provided.\n\nThe soft spot is the differential privacy guarantee. The mechanism clips each client's weight vector to an L2 norm bound and then adds Laplace noise with scale Δ/ε. But the Laplace mechanism requires L1 sensitivity. Two vectors with L2 norm ≤ Δ/2 can differ in L1 norm by up to Δ√d. For the 136-dimensional linear ranker on MSLR-WEB10K, the claimed ε=1.2 corresponds to an effective privacy budget of roughly 14. That is load-bearing because the thesis sells FPDGD as 'secured' and 'privacy-preserving'. The fact that Δ is chosen by grid search shows the sensitivity was not derived.\n\nA secondary concern: the unlearning verification uses a poisoning attack designed by the authors to test removal. That is self-referential, though common in the unlearning literature. I'd treat the unlearning results as promising but not conclusive.\n\nThe thesis is transparent about its limitations, and the writing is clear. I would give this a serious peer review, but the privacy section needs a major revision or a careful downscoping of the claim. For researchers working on federated ranking, this is a useful reference.\n\nRecommendation: send to peer review, with the expectation that the DP analysis will need to be reworked.","headline":"Empirical thesis on federated learning to rank with solid experiments, but the claimed ε-DP guarantee does not follow from the L2 clipping + Laplace noise mechanism.","tokens_in":52201,"tokens_out":4693,"would_cite":true,"duration_ms":44290,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This thesis claims that FPDGD, a federated adaptation of PDGD with differential privacy, is a reliable, stable, and secure federated online learning to rank method that significantly outperforms the only prior FOLTR baseline.","keywords":["federated learning","online learning to rank","differential privacy","non-IID data","poisoning attacks","machine unlearning","ranking effectiveness","privacy-preserving search"],"falsifier":"Measure the true global sensitivity of the FedAvg-aggregated PDGD update by computing the maximum L1 distance between aggregated updates obtained from two click datasets that differ in one client's interaction, and compare that value with the $\\Delta$ used in the privacy analysis; if the measured sensitivity exceeds $\\Delta$, the stated epsilon-differential privacy guarantee is invalid. A complementary test is to run a membership inference attack against the released local updates and check whether the empirical attack advantage matches what the claimed epsilon would permit.","tokens_in":51141,"feed_emoji":"🔍","tokens_out":3250,"duration_ms":126710,"temperature":0.7,"pith_summary":"The thesis tries to establish that online learning to rank can be moved into a federated, privacy-preserving setting without giving up ranking quality. Its central claim is that FPDGD, which trains rankers from implicit click feedback across distributed clients while sharing only weight updates, is effective, stable, and secure. Empirical evaluations are said to show FPDGD significantly outperforms the only prior federated OLTR method, FOLtR-ES, across datasets and click models. The thesis also claims to identify which non-IID data distributions actually hurt FOLTR, to show how poisoning attacks degrade it and which defenses help, and to supply an efficient unlearning method with a poison-based way to verify forgetting. If true, this would make privacy-preserving search ranking practical under real user behaviours, heterogeneous clients, and legal requirements to erase user contributions.","feed_headline":"Federated click-trained ranker beats prior FOLTR baseline","feed_subtitle":"A federated PDGD variant with differential privacy gains across datasets and stays stable under noisy clicks.","key_machinery":"The method that carries the argument is FPDGD: Federated Pairwise Differentiable Gradient Descent. It combines PDGD, which estimates a pairwise gradient from click preferences using a Plackett-Luce ranking model and inverse-propensity-style reweighting to reduce position bias, with the Federated Averaging algorithm, where each client runs local PDGD updates on a batch of queries and the server averages the returned weight vectors. Privacy is enforced by clipping each local update's norm to a bound $\\Delta$/2 and having each client contribute Gamma noise whose sum behaves as a Laplace random variable, implementing an epsilon-differential privacy mechanism.","core_discovery":"The central discovery is a new method, FPDGD, that casts the state-of-the-art centralised OLTR algorithm PDGD into the Federated Averaging framework and overlays an epsilon-differential privacy mechanism on the communicated updates. Each client performs local PDGD updates from clicks, clips the resulting weights to a norm bound, and adds Gamma noise that sums to a Laplace-distributed perturbation, so that the server never sees raw queries, documents, or clicks. The thesis reports that FPDGD consistently and significantly outperforms FOLtR-ES, the only prior FOLTR method, on large-scale datasets and under noisy click models, and that it remains stable when privacy budgets change. It further claims that only certain types of non-IID data, chiefly document preference skew and extreme label-distribution skew, seriously degrade FPDGD, and that standard non-IID remedies from general federated learning do not transfer, while data sharing helps the label-skew case. It also reports that data and model poisoning attacks can reduce ranking effectiveness and that robust aggregation rules such as Krum, Multi-Krum, Trimmed Mean, and Median provide defense, and finally that unlearning a client via historical local updates yields a ranker comparable to retraining from scratch.","pith_inferences":["The privacy guarantee is only as strong as the fitted clipping bound: because Delta is chosen by grid search rather than derived from PDGD's actual sensitivity, the reported epsilon values may understate the true privacy loss if the update's global sensitivity exceeds Delta.","The non-IID findings are tied to FPDGD's pairwise loss and FedAvg aggregation; other federated ranking methods, especially ones not using pairwise preferences, could respond differently to the same data distributions.","The poison-based unlearning verification is a transferable idea: a malicious-client probe can serve as a practical membership test for whether a federated model has erased a client's influence, beyond the ranking domain.","As federated rankers move toward pretrained-language-model architectures, the same attack, non-IID, and unlearning questions will need to be revisited because gradient statistics and update norms differ substantially from linear rankers."],"forward_implications":["FPDGD gives the first gradient-based federated online learning to rank method with reported consistent gains over the evolutionary-strategy baseline, making it a practical candidate for real federated search.","Adding differential privacy to FPDGD has little effect on ranking quality when enough clients participate, but it can badly hurt convergence when only a handful of clients are available.","Only some forms of non-IID data are dangerous for FPDGD: document-preference skew and single-label per-client skew degrade performance, while click-model variation and data-quantity skew do not.","Standard federated non-IID remedies such as FedProx and FedPer do not close the gap for Type 1 non-IID data; sharing a small global dataset helps for extreme label-distribution skew.","Untargeted data and model poisoning attacks can degrade FOLTR effectiveness, and robust aggregation rules mitigate them, with Krum behaving differently from Trimmed Mean and Median depending on attack strength and attack type."],"supporting_citations":[{"why":"Supplies the pairwise differentiable gradient descent method that FPDGD adapts to the federated setting.","marker":"[39]"},{"why":"Provides the Federated Averaging algorithm used to aggregate local ranker updates into the global model.","marker":"[7]"},{"why":"Introduces the prior FOLTR method FOLtR-ES, which serves as the main baseline and comparison point.","marker":"[27]"},{"why":"Establishes the reproducibility findings showing FOLtR-ES performs poorly on large datasets and standard OLTR metrics.","marker":"[1]"},{"why":"Contains the original FPDGD proposal with its empirical evaluation and privacy configuration.","marker":"[2]"},{"why":"Defines the Laplace mechanism for differential privacy that underlies the noise-adding procedure.","marker":"[60]"},{"why":"Provides the clipping technique for bounding update sensitivity in federated learning with differential privacy.","marker":"[62]"},{"why":"Shows how to generate a Laplace random variable from a sum of Gamma random variables, which the distributed noise scheme relies on.","marker":"[115]"}],"fun_headline_variants":["FPDGD: Federated PDGD with differential privacy outperforms FOLTR","Privacy-preserving federated ranker beat prior FOLTR in tests","Federated online ranker adds DP, beats baseline across datasets","New federated OLTR method with differential privacy tops prior","FPDGD: Privacy-safe federated learning to rank surpasses old"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The differential privacy guarantee holds only if clipping each local weight update to norm $\\Delta$/2 actually bounds the global sensitivity of the aggregated update by $\\Delta$; the paper selects $\\Delta$ by grid search rather than computing the true sensitivity.","fun_headline_variants_meta":{"raw":{"variants":["FPDGD: Federated PDGD with differential privacy outperforms FOLTR","Privacy-preserving federated ranker beat prior FOLTR in tests","Federated online ranker adds DP, beats baseline across datasets","New federated OLTR method with differential privacy tops prior","FPDGD: Privacy-safe federated learning to rank surpasses old"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000519,"raw_usage":{"total_tokens":2539,"prompt_tokens":997,"completion_tokens":1542,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":1448}},"tokens_in":613,"tokens_out":1542,"duration_ms":13408,"temperature":1.0,"reasoning_tokens":1448,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:57:13.604747+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the true global sensitivity of the FedAvg-aggregated PDGD update by computing the maximum L1 distance between aggregated updates obtained from two click datasets that differ in one client's interaction, and compare that value with the $\\Delta$ used in the privacy analysis; if the measured sensitivity exceeds $\\Delta$, the stated epsilon-differential privacy guarantee is invalid. A complementary test is to run a membership inference attack against the released local updates and check whether the empirical attack advantage matches what the claimed epsilon would permit.","supporting_citations":[{"cited_title":"Mugunthan, A","cited_arxiv_id":null,"evidence_quote":"Shows how to generate a Laplace random variable from a sum of Gamma random variables, which the distributed noise scheme relies on."}],"review_version":1}