{"id":"842d1300-c9bb-49a5-ab59-6c56729975c0","arxiv_id":"2501.04802","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"A reproducibility study speeds up HotFlip corpus poisoning 16x with query centroids and shows attack transfer is poor across retrievers while query-agnostic poisoning still hits Contriever.","lead":"This paper reproduces a gradient-based corpus poisoning attack on dense retrieval systems and cuts the time to generate each malicious passage from about 4 hours to 15 minutes. It also shows the attack transfers poorly across models but works without knowing user queries on some retrievers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equations 3–4 are algebraically exact for a fixed query batch, so the centroid 'optimization' is an implementation change; the effectiveness comparison is confounded by altered batch sampling and high variance, leaving 'without compromising performance' insufficiently supported.","rationale":"The reader identified Equations 3 and 4 as the weakest assumption, specifically that the centroid might not preserve ranking behavior. My analysis shows that for a fixed query batch, the mean-embedding objective is mathematically identical to the original objective under dot-product similarity: the average of dot products equals the dot product with the average query embedding. Therefore the concern about representativeness is not the true vulnerability. The load-bearing issue is that the paper compares two algorithms that differ not only in the centroid reformulation but also in how query batches are sampled and used across iterations. If the original resamples a batch each iteration and Ours uses a fixed precomputed centroid, then any performance difference is confounded with the change in stochasticity. The paper does not provide a controlled ablation isolating the efficiency gain from the objective change. The high variance reported for Ours at |A|=1 corroborates that the optimization changes the optimization dynamics. This does not invalidate the reported 16x speedup, for which the runtime measurements are concrete, but it weakens the secondary claim that performance is preserved without compromise. The conditional verdict remains appropriate: the authors should add the controlled equivalence test and report the batch size before the stronger claim is accepted. My verdict is unchanged from the reader's CONDITIONAL, hence UNCHANGED.","tokens_in":14916,"tokens_out":8413,"duration_ms":87846,"concrete_test":"Run both Reproduced and Ours with the same dataset, same random seed, same fixed query batch Qb (no resampling), and same passage initialization; if Equations 3–4 are an exact reformulation, the generated passages and final top-20 success rates should be identical up to floating point. Report the batch size used in the 4h vs 15min timing. Additionally, implement a 'sum-embedding baseline' that precomputes sum_q Eq(q) rather than the mean, while keeping the original per-iteration batch resampling, and compare its runtime and success to Ours. If the baseline matches Ours in speed and matches Reproduced in variance, the paper's attribution of the speedup to the centroid formulation is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the 16x speedup at equal attack quality via query-centroid embeddings (Equations 3 and 4). For dot-product similarity, (1/|Q|) * sum_q Eq(q)^T Ep(a) = (mean_q Eq(q))^T Ep(a), so replacing a fixed query set by its mean embedding is not an approximation; it is an exact rearrangement. Any effectiveness difference between Reproduced and Ours in Table 2 must therefore arise from changed stochasticity, e.g., resampling a fresh batch Qb each iteration in the original versus a fixed precomputed centroid, not from the centroid substitution itself. The paper does not control for this: it reports no ablation that keeps the original batch-sampling policy while precomputing the sum or mean of query embeddings, and it does not report batch size. The high variance for Ours at |A|=1 (e.g., 73.1±19.9 vs 80.1±2.7 for Contriever/NQ) shows the optimization changes the gradient-noise profile. Thus the speedup is credible, but the claim of preserving performance conflates an implementation optimization with a modeling change; Table 2 is not a controlled test of the efficiency/effectiveness trade-off.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reproduces the HotFlip-based corpus poisoning attack of Zhong et al. (2023) against dense retrievers. It first reports that the original results are largely reproducible, then proposes an optimization that replaces the query batch in the attack objective by the mean query embedding (Equations 3 and 4), reducing adversarial passage generation from about 4 GPU-hours to about 15 minutes per passage. It further extends the evaluation to transfer-based black-box attacks (RQ2) and query-agnostic attacks that use corpus passages instead of queries (RQ3). The experiments cover five to seven retrievers on NQ, MS MARCO, ArguAna, and FiQA, with five seeds and standard deviations. The main claimed outcomes are that the optimized pipeline preserves attack success while being much faster, that HotFlip does not transfer across retrieval models, and that query-agnostic attacks can be effective against some retrievers.","tokens_in":15145,"tokens_out":5078,"duration_ms":47804,"significance":"If the efficiency and effectiveness claims hold, the paper makes white-box HotFlip corpus poisoning practical at scale and provides a useful baseline for robust-retrieval research. The study has concrete strengths: the code is publicly released, the main experiments use five random seeds with reported standard deviations, the runtime comparison is concrete, and the derivation in Equations 3 and 4 is parameter-free. The RQ2 and RQ3 analyses add useful negative and positive evidence about HotFlip's generalization. However, the central 'without compromising performance' claim is not yet supported because the experimental comparison changes two variables at once—the centroid substitution and the batch-sampling policy—without a controlled ablation, and several reported differences are either large or statistically noisy. The paper is a solid reproduction study but needs an additional controlled experiment before the headline efficiency/effectiveness trade-off can be accepted.","major_comments":[{"comment":"The replacement of the query batch by its mean embedding is algebraically exact for dot-product similarity: (1/|Q|) sum_q E_q(q)^T E_p(a) = (mean_q E_q(q))^T E_p(a). Therefore this is an implementation optimization, not an approximation, and any effectiveness difference between Reproduced and Ours must be attributed to the changed stochasticity—in particular, the original resamples the query batch Q_b at each iteration while Ours precomputes a fixed centroid. The paper does not provide a controlled ablation that keeps the original batch-sampling policy while using the precomputed mean or sum, and it does not report the batch size. As a result, Table 2 does not support the claim 'without compromising performance'; it conflates the efficiency change with a change in gradient noise.","section":"Section 3.3, Eqs. (3)-(4)"},{"comment":"At |A|=1 on NQ, Ours reports 73.1±19.9 versus Reproduced 80.1±2.7, and on MS MARCO 56.3±13.4 versus 62.3±5.0; similar gaps appear for Contriever-ms. The paper describes these as 'slightly' underperforming, but the magnitudes and standard deviations indicate the difference is substantial and statistically noisy. With no significance tests and no effect-size reporting, the conclusion that performance is preserved is not established, especially for the models where the optimized pipeline performs worse.","section":"Table 2, Contriever and Contriever-ms rows"},{"comment":"The reproducibility claim is weakened by the large discrepancy for Contriever-ms at |A|=1 on NQ: Original reports 0.5 while Reproduced reports 48.4. The paper speculates about a possible typo in the original and notes that the authors did not respond, but it does not resolve which value is correct. Since the first contribution of the paper is verifying that the findings of Zhong et al. are reproducible, this unresolved discrepancy needs to be discussed more prominently and, if possible, checked against the original code and configuration.","section":"Section 5.1, Table 2"}],"minor_comments":[{"comment":"Please write '4 GPU-hours' (or '4 GPU·h') instead of '4 GPU/h'.","section":"Section 1"},{"comment":"Please report the query batch size used in Reproduced and Ours; it is needed to interpret the runtime and variance results.","section":"Section 4.3"},{"comment":"It is unclear whether the reported times are per adversarial passage or per generated set, and the figure does not show error bars; please clarify.","section":"Section 5.1, Figure 2"},{"comment":"The paper states that MS MARCO transfer results are omitted due to space limitations; given the strength of the transferability claim, at least a summary in an appendix would be helpful.","section":"Section 5.2"},{"comment":"The notation '≃' suggests approximation, but the equality is exact for a fixed batch; consider replacing it with '=' and instead explaining that the change is in the batch-sampling policy.","section":"Section 3.3, Eqs. (3)-(4)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the scope of the venue and the reproduction effort is valuable. The main revision should focus on the controlled batch-sampling ablation for Equations 3 and 4; if the authors can show the speedup at a matched sampling policy and report significance or effect sizes, the central claim would be much stronger. I do not see a circularity or integrity issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a credible reproduction study with a genuine 16x speedup and two new pieces of evidence (transfer failure, query-agnostic attacks), but the headline claim that the optimization preserves performance is not fully supported on the paper's own numbers. The reader's conditional accept is roughly right, and the stress-test point deserves attention.\n\nWhat's actually new: the centroid-based trick in Equations 3–4 is presented as an approximation, but for a fixed query batch and dot-product similarity it is algebraically exact. So the speedup is an implementation optimization—precomputing the mean embedding and skipping per-query gradient accumulation—not a new attack formulation. That's still useful. The reproduction of Zhong et al. is largely consistent; the authors report five seeds, release code, and are transparent about the Contriever-ms discrepancy (reproduced 48.4 vs original 0.5, flagged as a possible typo). The transfer matrix in Figure 3 is a clean negative result: HotFlip passages essentially don't transfer across models. The query-agnostic setup is a sensible extension, and the finding that 0.01% adversarial passages can hit Contriever hard is worth knowing.\n\nThe soft spots are in the effectiveness comparison. Because the centroid is exact for a fixed batch, any difference between Reproduced and Ours must come from changed stochasticity—the original resamples a query batch each iteration, while Ours uses a fixed precomputed mean. Without an ablation that keeps the original resampling policy, Table 2 is not a controlled test of the efficiency/effectiveness trade-off. The high variance at |A|=1 (Contriever on NQ: 73.1±19.9 vs 80.1±2.7) shows the gradient-noise profile changed, and the paper's own text admits Ours 'slightly underperforms' on Contriever and Contriever-ms. So 'without compromising performance' holds only on average and at larger |A|, not uniformly. The transfer and query-agnostic experiments need error bars and seed counts; as presented, the conclusions rest on unspecified repetitions. Also, the batch size is not reported.\n\nBottom line: this is a worthwhile contribution for adversarial IR researchers. It doesn't open a new line of inquiry, but it makes HotFlip practical and documents its limits. The math is fine, the code is a plus, and the limitations are mostly reporting issues rather than fatal flaws. I would send it to peer review with a request for major revision: add the resampling ablation, report batch size and error bars, and soften claims about 'consistently best' and 'without compromising performance.'","headline":"A solid reproducibility study with a real speedup and useful new evidence, but the 'without compromising performance' claim is muddied by a confounded comparison; worth peer review with revision.","tokens_in":15671,"tokens_out":4010,"would_cite":true,"duration_ms":36535,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that HotFlip corpus poisoning can be sped up about sixteen times, from four GPU-hours to fifteen minutes per passage, by targeting a precomputed query-centroid embedding, with no loss in top-20 attack success.","keywords":["corpus poisoning","dense retrieval","adversarial attacks","HotFlip","gradient-based word substitution","query-agnostic attack","transfer-based attack","embedding centroid"],"falsifier":"Run the optimized attack against a retriever or query distribution where the centroid is not representative, for example a deliberately bimodal query cluster whose members lie on opposite sides of the embedding space, and compare top-20 success against the original per-query HotFlip; a material drop in success rate would show the centroid approximation does not hold. A second check is to measure whether the 15-minute-per-document figure holds on a retriever outside the five tested, since the claim is tied to BERT-base-sized encoders.","tokens_in":14719,"feed_emoji":"🎯","tokens_out":5036,"duration_ms":43754,"temperature":0.7,"pith_summary":"This paper tries to establish two things: that the HotFlip corpus-poisoning attack on dense retrievers can be made roughly sixteen times faster without weakening it, and that the attack's real-world utility is narrower than the original report suggested. The speedup comes from replacing the per-query gradient accumulation with a single precomputed centroid embedding of a query batch, cutting generation from about four GPU-hours to about fifteen minutes per fifty-token adversarial passage while keeping top-20 attack success rates intact across five retrievers. The paper also shows that HotFlip passages do not transfer across different retrieval models, so black-box attacks via transfer mostly fail, and that query-agnostic attacks, which need no user queries at all, can still hit some retrievers with as little as 0.01% injected passages. A sympathetic reader would take away that the attack is now cheap enough to matter in practice, but its reach depends heavily on the target model.","feed_headline":"HotFlip corpus poisoning drops from 4 GPU-hours to 15 minutes","feed_subtitle":"A query-centroid trick keeps top-20 attack success while making white-box corpus poisoning practical at scale.","key_machinery":"The load-bearing object is the query-batch centroid embedding, $\\bar{E}_q(Q_b) = \\frac{1}{|Q_b|}\\sum_{q\\in Q_b} E_q(q)$, precomputed once per cluster of queries. HotFlip's token-substitution step normally approximates the effect of replacing token $t_i$ with $t'_i$ by the dot product $e_{t'_i}^\\top \\nabla_{e_{t_i}} \\mathrm{sim}(q,a)$ summed over queries; the paper instead uses the same approximation against the single centroid, $e_{t'_i}^\\top \\nabla_{e_{t_i}}(\\bar{E}_q(Q_b)^\\top E_p(a))$, so each search step needs one forward-backward pass instead of one per query. The same mechanism is re-used in the query-agnostic setting by swapping in the centroid of a batch of corpus passages, and the paper ties attack success on different retrievers to the $\\ell^2$ norm inflation of the generated adversarial passages.","core_discovery":"On the paper's own terms, the central discovery is that the objective of HotFlip corpus poisoning can be re-targeted at the mean embedding of a batch of queries without sacrificing attack quality. Equations 3 and 4 replace the sums over queries in the original objective with a single dot product against the precomputed centroid $\\bar{E}_q(Q_b)$, turning a process that needs gradients accumulated for every query-passage pair into one that needs a single gradient per token search. The reported result is a drop from an average of four GPU-hours to fifteen minutes per generated document on the same hardware, with top-20 success rates that match or exceed the reproduced original on several retrievers and only slight losses on Contriever and Contriever-ms. The paper further claims that the original HotFlip results are largely reproducible, with one notable discrepancy it attributes to a likely typo, and that the optimized version is also effective in a query-agnostic mode where passage centroids replace query centroids.","pith_inferences":["If the centroid approximation generalizes, the same trick could accelerate other gradient-based text attacks that average losses over input sets, such as document-manipulation or counterfactual generation, not just corpus poisoning.","The $\\ell^2$ norm correlation suggests a directly testable defense: at retrieval time, clip or penalize passage embedding norms before ranking; the paper does not test this, but its own histograms predict it would blunt attacks on Contriever-style models.","The speedup removes the main practical barrier to using HotFlip as a negative-example generator for robust training, so the authors' stated secondary use case could scale to millions of hard negatives.","Because the attack succeeds without queries on some retrievers, privacy-minded systems cannot rely on query secrecy alone; defenders need model-level robustness or norm-based detection."],"forward_implications":["A white-box attacker can now poison a dense-retrieval corpus at a cost of about 15 minutes per adversarial document on an NVIDIA L40, making large-scale corpus contamination feasible rather than theoretical.","HotFlip's lack of transferability across retrieval models means a defender can ignore cross-model poisoning unless the attacker has access to the exact target encoder.","Query-agnostic poisoning, needing no user queries, is a realistic threat for retrievers like Contriever and Contriever-ms, where 0.01% injected passages already exceed 80% top-20 success.","Because adversarial passages from high-norm retrievers show a large $\\ell^2$ norm increase over normal passages, norm-based filtering is a plausible cheap defense.","The suspected typo in the original reported result means published attack numbers should be read with caution, even when the method itself is sound."],"supporting_citations":[{"why":"Supplies the original HotFlip corpus-poisoning pipeline that this paper reproduces and optimizes.","marker":"[34]"},{"why":"Defines HotFlip, the gradient-based token-substitution method whose efficiency the paper improves.","marker":"[5]"},{"why":"Provides the Contriever models used as attack targets and as source retrievers.","marker":"[9]"},{"why":"Provides the DPR-nq and DPR-mul dense retrievers used in the attack evaluation.","marker":"[10]"},{"why":"Provides the ANCE retriever used in the attack evaluation.","marker":"[31]"},{"why":"Supplies the NQ dataset used for in-domain and transfer attacks.","marker":"[11]"},{"why":"Supplies the MS MARCO dataset used for in-domain attacks.","marker":"[1]"},{"why":"Supplies the BEIR benchmark from which ArguAna and FiQA are taken for query-agnostic attacks.","marker":"[26]"},{"why":"Documents HotFlip's computational inefficiency, the motivation for the optimization.","marker":"[35]"},{"why":"Provides AGGD, a comparable corpus-poisoning method with a different token-selection strategy, used as context.","marker":"[24]"}],"fun_headline_variants":["HotFlip poisoning: 16x faster via query-centroid trick","Corpus poisoning attack scaled: HotFlip now 16x faster","Query-centroid trick accelerates HotFlip poisoning 16-fold","HotFlip attacks dense retrievers 16x faster with same success","Efficient HotFlip: corpus poisoning in 15 minutes per doc"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole speedup rests on the assumption that the mean embedding of a batch of queries preserves the ranking behaviour of the full query set, so that optimizing against the centroid is equivalent to optimizing the original HotFlip objective.","fun_headline_variants_meta":{"raw":{"variants":["HotFlip poisoning: 16x faster via query-centroid trick","Corpus poisoning attack scaled: HotFlip now 16x faster","Query-centroid trick accelerates HotFlip poisoning 16-fold","HotFlip attacks dense retrievers 16x faster with same success","Efficient HotFlip: corpus poisoning in 15 minutes per doc"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00047,"raw_usage":{"total_tokens":2366,"prompt_tokens":997,"completion_tokens":1369,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":1280}},"tokens_in":613,"tokens_out":1369,"duration_ms":10068,"temperature":1.0,"reasoning_tokens":1280,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:24:44.697189+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the optimized attack against a retriever or query distribution where the centroid is not representative, for example a deliberately bimodal query cluster whose members lie on opposite sides of the embedding space, and compare top-20 success against the original per-query HotFlip; a material drop in success rate would show the centroid approximation does not hold. A second check is to measure whether the 15-minute-per-document figure holds on a retriever outside the five tested, since the claim is tied to BERT-base-sized encoders.","supporting_citations":[{"cited_title":"In: Gurevych, I., Miyao, Y","cited_arxiv_id":null,"evidence_quote":"Defines HotFlip, the gradient-based token-substitution method whose efficiency the paper improves."},{"cited_title":"In: 9th International Conference on Learning Representa- tions, ICLR 2021, Virtual Event, Austria, May 3-7, 2021","cited_arxiv_id":null,"evidence_quote":"Provides the ANCE retriever used in the attack evaluation."}],"review_version":1}