{"id":"feed0f13-059b-4961-acf8-0b35c0942cea","arxiv_id":"2411.14937","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Geminio uses a vision-language model to reshape a federated model so that gradients from a victim client reveal only the images that match an attacker's text query.","lead":"This paper shows a federated learning server can use natural language descriptions to target and reconstruct specific private images, such as faces or weapons, from gradients a client shares. It matters because it turns a known privacy attack into a flexible, semantic tool that works on large batches and evades common defenses.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The attack mechanism hinges on an unproved proportionality between per-sample gradient norm and per-sample loss; if this fails for the crafted model, Geminio degenerates to a label-poisoning baseline, so the transfer must be measured directly.","rationale":"The reader identifies the same weakest assumption: the attack's central mechanism transfers per-sample loss amplification to gradient dominance via an unproved proportionality. This is indeed the load-bearing step; if it fails, the crafted model cannot steer batch-averaged gradients toward query-matched samples, so any reconstruction optimizer would recover the wrong data, and the method reduces to a label-poisoning baseline. The paper asserts the proportionality without evidence, and the experiments measure downstream success rather than isolating this transfer. I therefore agree with the reader's assessment and recommend keeping the conditional verdict: the claim is plausible and the qualitative reconstructions are promising, but the mechanism needs direct validation before the attack can be accepted as described. I additionally note that Equation (4) as printed appears to be constant (the numerator and denominator are the same sum up to the factor |Baux|), which would make the training objective vacuous; this is likely a typographical error, but it should be corrected in a revision, since it further undermines reproducibility of the exact method.","tokens_in":16909,"tokens_out":9692,"duration_ms":96333,"concrete_test":"Implement Equation (4) literally as written and run one optimization step on a random model; if the loss is identically 1/|Baux| with zero gradient, the printed training objective is vacuous. Then inspect the released code to identify the actual loss used; if it differs, the paper must be corrected.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 states: 'we could exploit the property that the per-sample gradient magnitude is proportional to the per-sample loss value' and uses this to assert that the crafted model's gradients will be dominated by query-matched samples. This is not generally true: for cross-entropy with saturated softmax, gradient norm can be small while loss is large, and per-layer parameter gradients depend on activations and local derivatives, not just scalar loss. The entire pipeline—malicious model transfer to any reconstruction optimizer—rests on this transfer. If the trained model produces high losses on matched samples but their gradients are not systematically larger than those of unmatched samples, the submitted batch gradient will not be dominated by the target, and reconstruction fails. The paper provides no measurement of per-sample gradient norms on the malicious model, only end-to-end attack metrics that may reflect other factors (e.g., label poisoning). This is the weakest link in the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Geminio, an attack in which a malicious FL server crafts a global model, guided by a pretrained vision-language model and unlabeled auxiliary data, so that when a victim client trains on it, the per-sample losses and hence gradients of images matching a natural-language query dominate the batch-averaged gradient. Existing gradient inversion optimizers can then be applied to reconstruct those targeted images from large batches. The paper reports qualitative reconstructions on ImageNet, FER, and CIFAR-20, quantitative attack recall/precision on CIFAR-20, plugin results with DLG, InvertingGrad, and HFGradInv, and robustness experiments under FedAvg, gradient pruning, Laplacian noise, parameter inspection, loss inspection, and homomorphic encryption.","tokens_in":17079,"tokens_out":3902,"duration_ms":38246,"significance":"If correct, Geminio is a significant advance for gradient inversion attacks: it gives the adversary flexible, semantically meaningful control over which private samples are recovered, works as a plugin to existing reconstruction optimizers, does not modify the neural architecture, and reportedly survives several defenses. The paper's strengths include public source code, experiments across multiple datasets, architectures, FL protocols, and defense mechanisms, comparison with a label-poisoning baseline, and ablation of auxiliary data and VLM choices. The main weaknesses are that the headline quantitative metric measures gradient dominance rather than actual image reconstruction, and the central gradient-loss proportionality assumption is asserted without direct verification.","major_comments":[{"comment":"The attack recall and precision metrics define success as a per-sample output-layer gradient having cosine similarity at least 0.90 with the batch-averaged gradient, not as actual image reconstruction. The threshold is chosen after observing that Fishing's 0.95 is 'overly restrictive,' so the headline numbers are partly post hoc. Since the paper's claim is that Geminio enables targeted reconstruction, the reported 64.96% recall and 65.67% precision for batch size 256 do not by themselves establish reconstruction. Please report reconstruction-grounded success for these settings (e.g., LPIPS against ground-truth matched images, or agreement with the query by a detector/captioning model) and/or clearly separate the 'gradient dominance' claim from the 'reconstruction' claim.","section":"Section 4.1 and Appendix E.1"},{"comment":"The mechanism assumes that the per-sample gradient norm is proportional to the per-sample loss value. This is not generally true: under cross-entropy with a saturated softmax, a sample can have high loss but small gradients, and per-layer gradient norms depend on activations and local derivatives, not just on the scalar loss. The entire transfer from the crafted model to any reconstruction optimizer rests on this assumption. Please add a direct measurement: on the malicious model, compare per-sample gradient norms of query-matched versus unmatched samples, ideally per layer and across batch sizes, and show that dominance holds; also verify that the reconstructed images are actually the matched samples rather than artifacts of label poisoning.","section":"Section 3, paragraph after Eq. (4)"},{"comment":"The claim that Geminio 'has no impact on normal training' and that the server can 'still produce a high-utility ML model' is not supported by any reported utility measurement, such as the final global model's test accuracy compared to benign FL. Because the malicious server sends a randomly initialized or adversarially reshaped model to the victim and uses only non-victim gradients for aggregation, the utility question is nontrivial, especially at early rounds. Please report the global model's test accuracy over FL rounds for Geminio versus benign training.","section":"Section 4.3 and Appendix E"}],"minor_comments":[{"comment":"The statement that 'we observe multiple examples where targeted reconstruction succeeds even if the cosine similarity is below 0.90' undermines the chosen threshold; please either justify the threshold theoretically or report how many successful reconstructions fall below it.","section":"Appendix E.1"},{"comment":"The text refers to 'Geminio-adaptive in Figure 5,' but Figure 5 does not appear to display this variant; please add the figure or correct the pointer.","section":"Section 4.3, per-sample loss inspection"},{"comment":"Some references are malformed, for example [18] 'Yuxin Wen Jonas Geiping' and [29] 'Aidmar Wainakh...'; please normalize the bibliography entries.","section":"References"},{"comment":"The figure reports maximum parameter magnitudes of 2772.89 and 1000 for Fishing and GradFilt without specifying units or which layer; please clarify whether these are absolute maximum weight values and explain how GradFilt produces a parameter of magnitude 1000.","section":"Figure 13"}],"recommendation":"major_revision","confidential_remarks":"This is a promising and topical paper with a clear novelty claim, but the quantitative core needs strengthening before I can support acceptance. I would ask the authors to directly measure gradient dominance on the malicious model, report reconstruction-grounded success metrics, and provide global-model utility numbers. If those experiments confirm the current claims, the paper would be a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nGeminio is worth your time: it's the first attack I've seen that lets an FL server specify a natural-language query and then recover matching images from a large victim batch. The idea is to use CLIP to train a malicious global model whose loss surface amplifies only query-relevant samples, so the victim's batch gradients are dominated by those samples. The paper shows this works across ImageNet, CIFAR-20, and FER, with several architectures, and it can be bolted onto existing gradient-inversion optimizers like HFGradInv. The qualitative reconstructions (weapons, faces, a safe) are convincing. Code is available.\n\nWhat's new here is real: prior targeted attacks were class-level (Fishing, GradFilt) or used hand-crafted conditions (brightness, outliers). A language interface is a genuine step forward, and the soft-label trick for unlabeled auxiliary data is practical.\n\nThe soft spots are mostly in the evaluation, not the core idea. The headline recall/precision numbers are computed by checking whether a sample's per-image gradient is within a cosine similarity threshold of the batch average—they are not measuring actual successful reconstruction. That's a proxy, and the threshold (0.90) was chosen after the fact because Fishing used 0.95. The paper does show LPIPS for some reconstruction experiments, but the quantitative attack metrics should be replaced by or supplemented with actual reconstruction-quality metrics. I also note no error bars anywhere, which matters when comparing methods. The claim that Geminio has 'no impact on normal training' is never measured: the server only aggregates gradients from non-victim clients, so utility should be identical, but they don't show that.\n\nThe stress-test note about per-sample gradient norm not being strictly proportional to loss is a fair theoretical caveat, but the end-to-end reconstructions demonstrate the mechanism actually works in the tested settings. It would be good to see a direct measurement of per-sample gradient norms on the malicious model to verify the claim, but I don't see it as a load-bearing flaw.\n\nDefense evaluation is thin: gradient pruning, Laplacian noise, parameter inspection. No DP, no secure aggregation (though they have a short appendix note for homomorphic encryption). That's fine for a first attack paper.\n\nBottom line: this is a solid, incremental-but-meaningful contribution to the attack literature. It deserves a serious referee and likely publication after revisions that tighten the metrics and add error bars. I'd bring it to reading group and would cite it in my own work on FL privacy.","headline":"Genuinely new language-guided gradient inversion attack; works in experiments, but quantitative metrics measure gradient dominance, not reconstruction, and utility claim is unverified.","tokens_in":17620,"tokens_out":2237,"would_cite":true,"duration_ms":21126,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A malicious federated-learning server can use a text query to steer gradient inversion and recover only the private images that match, even from large batches.","keywords":["gradient inversion attacks","federated learning","vision-language models","natural language query","targeted data reconstruction","loss landscape reshaping","privacy attacks","CLIP"],"falsifier":"Take a query-matched image whose classification logits the poisoned model pushes far negative, making its cross-entropy loss high but its final-layer gradient near zero; if the victim-submitted gradient is then dominated by another sample, the proportional-loss premise fails and targeted reconstruction should collapse.","tokens_in":16723,"feed_emoji":"🕵️","tokens_out":6043,"duration_ms":48901,"temperature":0.7,"pith_summary":"The paper claims that a malicious federated-learning server can turn gradient inversion into a targeted search: the attacker types a natural-language query, and the server recovers only the private images that match it, even when the victim trained on a large batch. If true, this erodes a core privacy promise of federated learning—that shared gradients reveal little about individual samples—because the attack runs in any round and does not degrade the utility of the global model. The key move is to poison the global model so that query-matched samples produce amplified losses and therefore dominate the gradients the victim returns, which any existing gradient-inversion optimizer can then exploit.","feed_headline":"Text queries let a malicious server pick which private images to steal","feed_subtitle":"Geminio reshapes model losses so matching images dominate shared gradients, beating large-batch defenses.","key_machinery":"The load-bearing object is the reshaped loss landscape. Geminio minimizes a ratio loss in which each auxiliary sample's cross-entropy term is scaled by $1-\\alpha(x; Q, B_{aux})$, with $\\alpha$ a softmax over VLM query-image similarities computed across the auxiliary batch. Minimizing this ratio forces the model to suppress loss on query-irrelevant samples, and the batch-wise normalization correspondingly raises the relative loss of matched samples; the pretrained VLM (CLIP) supplies both the query similarity scores and, through class-name similarities, the soft labels that make the auxiliary dataset unlabeled.","core_discovery":"The central discovery is that a pretrained vision-language model (VLM) such as CLIP can be repurposed as a supervisor for crafting a malicious global model whose loss surface has a narrow peak on samples semantically similar to the attacker's query. Geminio trains this model on an unlabeled auxiliary dataset, using VLM similarity scores to assign soft labels and to weight each sample's loss by $1-\\alpha$ so that query-irrelevant samples are driven to near-zero loss while matched samples keep high loss. Because per-sample gradient magnitude is assumed proportional to per-sample loss, the victim's batch-averaged gradient ends up dominated by matched samples, and existing reconstruction optimizers recover those samples with high fidelity. The attack is task-agnostic, works at any FL round, and leaves the global model's utility unchanged because the server aggregates gradients from non-victim clients.","pith_inferences":["Beyond the paper's experiments, the same VLM-guided loss-reshaping recipe could generalize to other data modalities such as audio, time series, or tabular records wherever a text-embedding similarity signal exists.","A direct test of the mechanism would compare per-sample gradient norms against per-sample losses on the poisoned model; architectures or losses where high loss does not yield high gradient norm would likely break the attack.","The results imply that FL servers should be treated as capable of targeted data retrieval, shifting defense research toward detecting unusual per-sample loss distributions or training dynamics rather than parameter magnitudes."],"forward_implications":["Any existing gradient-inversion optimizer (HFGradInv, InvertingGrad, DLG) can be upgraded to a targeted attack by consuming Geminio-poisoned gradients, recovering recognizable images from batches of 64–256.","On CIFAR-20, attack recall stays near 65% at a victim batch size of 256, while vanilla inversion becomes ineffective beyond batch size 8.","Geminio works under FedAvg when the server assigns a small learning rate, and it also boosts the accuracy of class-label inference, not just image reconstruction.","Standard defenses such as gradient pruning, Laplacian noise, and parameter-magnitude inspection fail to stop the attack at settings that preserve model utility."],"supporting_citations":[{"why":"Supplies the pretrained VLM whose image–text similarities score query relevance and generate soft labels for unlabeled auxiliary data.","marker":"[24]"},{"why":"Provides the default reconstruction optimizer, HFGradInv, that consumes the poisoned gradients and recovers targeted images.","marker":"[36]"},{"why":"Provides InvertingGrad, the gradient-inversion baseline and reconstruction approach that Geminio enhances.","marker":"[12]"},{"why":"Provides DLG, the early gradient-inversion method shown to become capable of high-resolution targeted recovery when fed Geminio gradients.","marker":"[47]"},{"why":"Introduces gradient magnification and the cosine-similarity success metric used to evaluate attack recall and precision.","marker":"[35]"},{"why":"Demonstrates natural-language-guided model poisoning in FL, the design template Geminio adapts from backdoors to inversion attacks.","marker":"[3]"}],"fun_headline_variants":["Geminio: text queries target private image theft in federated learning","Language-guided attacks let servers steal chosen images from clients","Geminio: natural language steers gradient inversion to specific data","Server whispers a query, Geminio steals matching private images"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attack hinges on the assumption that a sample's gradient magnitude is proportional to its loss value, so making matched samples' losses large makes their gradients dominate the batch average.","fun_headline_variants_meta":{"raw":{"variants":["Geminio: text queries target private image theft in federated learning","Language-guided attacks let servers steal chosen images from clients","Geminio: natural language steers gradient inversion to specific data","Server whispers a query, Geminio steals matching private images"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00029,"raw_usage":{"total_tokens":1728,"prompt_tokens":1007,"completion_tokens":721,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":649}},"tokens_in":623,"tokens_out":721,"duration_ms":6946,"temperature":1.0,"reasoning_tokens":649,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:41:23.956555+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a query-matched image whose classification logits the poisoned model pushes far negative, making its cross-entropy loss high but its final-layer gradient near zero; if the victim-submitted gradient is then dominated by another sample, the proportional-loss premise fails and targeted reconstruction should collapse.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained VLM whose image–text similarities score query relevance and generate soft labels for unlabeled auxiliary data."},{"cited_title":"High- fidelity gradient inversion in distributed learning","cited_arxiv_id":null,"evidence_quote":"Provides the default reconstruction optimizer, HFGradInv, that consumes the poisoned gradients and recovers targeted images."},{"cited_title":"Inverting gradients-how easy is it to break privacy in federated learning? Advances in neural informa- tion processing systems, 33:16937–16947, 2020","cited_arxiv_id":null,"evidence_quote":"Provides InvertingGrad, the gradient-inversion baseline and reconstruction approach that Geminio enhances."},{"cited_title":"dinosaur,","cited_arxiv_id":null,"evidence_quote":"Provides DLG, the early gradient-inversion method shown to become capable of high-resolution targeted recovery when fed Geminio gradients."},{"cited_title":"Fishing for user data in large- batch federated learning via gradient magnification","cited_arxiv_id":null,"evidence_quote":"Introduces gradient magnification and the cosine-similarity success metric used to evaluate attack recall and precision."},{"cited_title":"Imperio: Language- guided backdoor attacks for arbitrary model control","cited_arxiv_id":null,"evidence_quote":"Demonstrates natural-language-guided model poisoning in FL, the design template Geminio adapts from backdoors to inversion attacks."}],"review_version":1}