Pith. sign in

REVIEW 2 major objections 6 minor 37 references

Targeted Mismatch Adversarial Attack: Query with a Flower to Retrieve the Tower

T0 review · 2 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adversarial flower images conceal a query but retrieve the same results.

desk verdict A solid, genuinely new attack for CNN retrieval with honest ablations; the privacy claim needs per-query statistics but the core contribution holds. read the letter →

arxiv 1908.09163 v1 pith:WNYUMVXO submitted 2019-08-24 cs.CV

Giorgos Tolias , Filip Radenovic , Ondřej Chum This is my paper · ORCID
classification cs.CV
keywords targetedmismatchattackadversarialimageretrievalprivacyCNNdescriptorglobalpoolingactivationhistogramqueryconcealment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces the targeted mismatch attack, a way to hide the image a user actually wants to search for by submitting a visually unrelated adversarial image instead. If the attack works, a user can ask an online image search engine whether a private or copyrighted photo is indexed without ever uploading that photo. The authors formulate the attack as an optimization problem and design loss functions that survive unknown global pooling, unknown input resolution, and unknown whitening in the retrieval system, as long as the feature-extraction network is known. They demonstrate on standard retrieval benchmarks that the adversarial image yields near-identical rankings to the original query.

What carries the argument

The central object is the targeted mismatch attack, an optimization over the adversarial image $x$ of $L_{\mathrm{tr}}(x_c, x_t; x) = \ell_{\mathrm{tr}}(x, x_t) + \lambda \|x - x_c\|^2$, where $x_c$ is the carrier image, $x_t$ is the target, and $\ell_{\mathrm{tr}}$ is one of three performance losses: descriptor matching, activation-tensor matching, or activation-histogram matching with soft RBF bin assignment. Multi-scale and Gaussian-blurred variants of these losses provide robustness to unknown test resolution and down-sampling behavior. The optimization is initialized from the carrier and run with Adam; the authors find that setting $\lambda = 0$ works because the optimizer converges to local minima that stay visually close to the carrier.

What would settle it

Run the attack with $\lambda=0$ for a target photo and a carrier that is visually similar to the target, such as the same building from a slightly different angle; if human judges identify the optimized image as the target more often than the carrier, the non-disclosure property is falsified.

Watch

Extended reading notes

Core claim

The central claim is that targeted mismatch attacks are practical: given a carrier image and a target image, minimizing a loss that combines a retrieval-performance term with a distortion term produces an adversarial image that looks like the carrier, is visually dissimilar to the target, and retrieves the same results. Matching channel-wise activation histograms, rather than the full activation tensor or a single descriptor, is the key to being robust to unknown pooling while avoiding spatial leakage of the target's content.

Load-bearing premise

The privacy guarantee rests on the empirical observation that, even with $\lambda=0$, initializing from the carrier image makes the optimizer converge to a local minimum that looks like the carrier rather than the target; if a different network or optimizer breaks that behavior, the generated image could reveal the target.

Editorial extensions

If this is right

  • A user can check whether a private or copyrighted image is present in a search index without uploading the original image.
  • The attack succeeds when only the feature extractor is known and the pooling, resolution, or whitening are partially unknown.
  • Activation-histogram matching prevents the target's spatial layout from leaking into the adversarial image, whereas full-tensor matching allows the target to be reconstructed by inversion.
  • Transferring the attack to an entirely unseen feature extractor is much harder than for classification attacks, limiting the threat to systems whose backbone is known.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The empirical finding that $\lambda = 0$ stays near the carrier suggests a broader design rule: in feature-matching attacks, initialization and optimizer choice can enforce a perceptual constraint without an explicit distortion penalty.
  • The histogram-matching objective could be reused outside retrieval, for example in privacy-preserving comparisons of neural features where only per-channel statistics are released.
  • A boundary check worth testing: whether a spatial-order-aware pooling operation, such as a learned attention layer, breaks the invariance that histogram matching claims over global pooling.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper introduces the targeted mismatch attack for CNN-based image retrieval. Given a target query image and a visually different carrier image, the method optimizes an adversarial image that produces a descriptor highly similar to the target's descriptor while remaining visually close to the carrier, so that a retrieval system returns the same results as for the target without disclosing the target image. The paper proposes several loss functions: on the global descriptor, on the activation tensor, and on channel-wise activation histograms, with extensions for unknown pooling, unknown input resolution (via multi-scale optimization and Gaussian blurring), and ensembles over networks. Experiments on Holidays, Copydays, ROxford, and RParis with AlexNet, ResNet18, and VGG16 show that the attacks preserve retrieval quality for known and partially unknown systems, and that tensor inversion does not reveal the target for the histogram-based attacks.

Significance. If the claims hold, this is the first targeted attack on CNN retrieval and a practical method for privacy-preserving queries in visual search. The paper provides extensive empirical validation across four benchmarks, multiple networks, ablations over pooling, resolution, blurring, and whitening, and releases code. The attack is shown to be robust to partially unknown systems (pooling and resolution), which is a realistic threat model. However, the non-disclosure property, which is central to the privacy claim, relies on an unquantified empirical observation about the optimization landscape, and the visual dissimilarity to the target is only demonstrated through a few examples. With additional quantitative evidence on the distribution of query outcomes, the paper would fully support its central claim.

major comments (2)
  1. [Section 5.2 and Eq. (7)] The non-disclosure property rests on the statement that with λ=0 and carrier initialization, Adam converges to local minima “significantly closer to xc than xt.” No quantitative support is provided for this across the 50/70 queries. Tables 1 and 2 report only mean similarity to the target and mAP differences; similarity to the carrier and the worst-case behavior are never reported. Since Eq. (7) with λ=0 has no term preventing convergence to xt, a nontrivial fraction of queries converging near the target would invalidate the abstract's claim that “the generated image looks nothing like the user intended query.” Please provide per-query statistics (e.g., histograms or quantiles of descriptor similarity/distance to both carrier and target) for all attack configurations, and report the number or fraction of queries where the adversarial image is more similar to the target than to the carrier.
  2. [Section 4.1 and Figure 7] Visual dissimilarity to the target is modeled as descriptor similarity to the carrier, but no quantitative perceptual evaluation is performed. The paper shows a handful of qualitative examples (Figures 3, 6, and 7) and reports descriptor similarities; however, the central privacy claim is about human perception. A descriptor being close to the carrier does not guarantee perceptual dissimilarity from the target. I recommend reporting a perceptual similarity metric (e.g., LPIPS) or a small user study on the full query set, or at least on a random subset, to substantiate “looks nothing like.”
minor comments (6)
  1. [Section 4.2, Eq. (10)] The statement “It guarantees identical descriptors for all global pooling operations that ignore spatial information” is too strong. The implementation uses soft-assignment histograms (footnote 2), so exact histogram equality is not reached by the optimization; the appropriate wording is approximate preservation or a formal bound.
  2. [Section 5.3, “Impact of the distortion term”] The study of λ uses only RParis and one attack variant; reporting the same analysis on at least one other dataset would strengthen the conclusion that λ=0 is safe.
  3. [Figures 3 and 4] The curves are for a single target-carrier pair; adding error bars or multiple runs would show variability across queries.
  4. [Section 5.1] The choice to use only the first 50 queries for Holidays and Copydays should be justified (e.g., computational cost) and its effect on statistical significance noted.
  5. [Section 1, Introduction] “An non-targeted attack” should be “A non-targeted attack.”
  6. [Table 2] The “mAP difference to original” is sometimes positive and sometimes negative; a short note clarifying that positive values mean the attack improves retrieval would help avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: attack success on fully known test-models reflects the optimization objective by construction, but the paper's novel robustness and privacy claims are tested on held-out configurations and rest on empirical observations, not definitional equivalence.

full rationale

The paper derives no hidden prediction from first principles. Its central mechanism is an optimization: the adversarial image is constructed to minimize a loss that matches either the target's global descriptor, activation tensor, or activation histograms. For the fully known test-model, high descriptor similarity and matched retrieval results are therefore consequences of the objective, and the paper does not present these as independent predictions; it provides them as evidence that the optimizer works. The claimed novelty lies in partially unknown systems: unknown pooling, unknown test resolution, and unknown whitening. Those test configurations are not included in the optimization (e.g., test resolutions outside the attack-resolution set, pooling operations not directly optimized, whitening ignored during optimization), so the transfer results are externally grounded rather than forced by the loss. The privacy/non-disclosure claim is weaker: it depends on the empirical observation in Section 5.2 that with lambda=0 and carrier initialization, Adam converges to local minima closer to the carrier than to the target. That observation is not a derivation, and it is not circular — it is an unquantified empirical regularity, closer to a robustness limitation than to a circular step. Self-citations to the authors' prior work (R-MAC, ROxford/RParis, fine-tuning) are used only for standard network components and benchmarks, not as load-bearing justifications for the attack's validity. No fitted parameter is renamed as a prediction, and no cited uniqueness theorem or ansatz is smuggled in via self-citation. Thus the paper is self-contained with respect to its claims, and the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or conceptual entity; it constructs adversarial images through standard optimization. The main assumptions are domain-level (global-descriptor retrieval, white-box FCN) plus empirical convergence behavior for concealment. No free parameters are fitted to the target retrieval results.

free parameters (4)
  • Distortion weight lambda = 0 (default), experiments at 0.1, 1, 10
    Trade-off between descriptor matching and visual proximity to the carrier. Hand-chosen, swept in experiments; not fitted to data.
  • GeM pooling exponent = 3
    Fixed for all GeM-based losses; standard choice in retrieval literature, not fitted here.
  • Histogram RBF kernel width sigma_hist = 0.1
    Bandwidth for soft histogram assignment; hand-chosen, affects the approximate guarantee of the histogram loss.
  • Blur kernel width sigma_b = 0.3 * max(W,H) / s
    Hand-chosen heuristic to reduce sensitivity to different downsampling implementations.
assumptions (5)
  • domain assumption Retrieval is performed by nearest neighbor search on L2-normalized global descriptors via inner product.
    Section 3.2 defines the image retrieval model; the attack inherits this assumption. If the system used local descriptors or re-ranking, matching descriptors would not imply identical results.
  • domain assumption White-box access to the FCN feature extractor.
    Section 4.2 states 'we assume a white-box access to the FCN'. The attack optimizes against the FCN's activations; transfer to unseen FCNs is shown to be hard.
  • domain assumption Optimization from the carrier with lambda = 0 converges to a local minimum near the carrier, not the target.
    Section 5.2 reports this observation; the non-disclosure property depends on it. It is empirical, not proven.
  • domain assumption Channel-wise activation histograms determine the output of any spatial-invariant global pooling.
    Section 4.2 claims this as a guarantee; mathematically true for exact histograms, but the implementation uses soft RBF histograms with finite bins, so it holds only approximately.
  • standard math Adam optimization with projected gradient descent finds a minimum of the composite loss.
    Section 4.3; standard optimization, no special guarantee required for the empirical claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Targeted Mismatch Adversarial Attack: Query with a Flower to Retrieve the Tower." pith.science (2026). https://pith.science/paper/WNYUMVXO

@misc{pith2026190809163,
  author       = {Pith},
  title        = {Pith review of: Targeted Mismatch Adversarial Attack: Query with a Flower to Retrieve the Tower},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WNYUMVXO}},
  note         = {Machine review of arXiv:1908.09163}
}
read the original abstract

Access to online visual search engines implies sharing of private user content - the query images. We introduce the concept of targeted mismatch attack for deep learning based retrieval systems to generate an adversarial image to conceal the query image. The generated image looks nothing like the user intended query, but leads to identical or very similar retrieval results. Transferring attacks to fully unseen networks is challenging. We show successful attacks to partially unknown systems, by designing various loss functions for the adversarial image construction. These include loss functions, for example, for unknown global pooling operation or unknown input resolution by the retrieval system. We evaluate the attacks on standard retrieval benchmarks and compare the results retrieved with the original and adversarial image.

Figures

Figures reproduced from arXiv: 1908.09163 by the authors.

Figure 1
Figure 1. Top two rows show retrieval results to the user query [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. In targeted mismatch attacks an adversarial image is gen [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Adversarial images are generated with different loss functions and various measurements are reported as they evolve with the [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Descriptor similarity between the adversarial image and [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Performance evaluation for attack based on AlexNet and a set of attack-resolutions. Mean average Precision on [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Adversarial examples for a carrier image and two dif [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Target, carrier and adversarial images for different variants (top image row), a summary of tensor [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 32 canonical work pages

  1. [1]

    Threat of adversarial at- tacks on deep learning in computer vision: A survey

    Naveed Akhtar and Ajmal Mian. Threat of adversarial at- tacks on deep learning in computer vision: A survey. IEEE Access, 2018. 2

    work page 2018

  2. [2]

    The vulnerability of learning to adversarial perturbation increases with intrinsic dimensionality

    Laurent Amsaleg, James Bailey, Dominique Barbe, Sarah Erfani, Michael E Houle, Vinh Nguyen, and Milo ˇs Radovanovi´c. The vulnerability of learning to adversarial perturbation increases with intrinsic dimensionality. InIEEE Workshop on Information F orensics and Security (WIFS) ,

    work page

  3. [3]

    NetVLAD: CNN architecture for weakly supervised place recognition

    Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pa- jdla, and Josef Sivic. NetVLAD: CNN architecture for weakly supervised place recognition. In CVPR, 2016. 3

    work page 2016

  4. [4]

    Aggregating deep convolutional features for image retrieval

    Artem Babenko and Victor Lempitsky. Aggregating deep convolutional features for image retrieval. In ICCV, 2015. 3

    work page 2015

  5. [5]

    A limited memory algorithm for bound constrained optimization

    Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained optimization. SISC, 1995. 2

    work page 1995

  6. [6]

    Hidden voice commands

    Nicholas Carlini, Pratyush Mishra, Tavish Vaidya, Yuankai Zhang, Micah Sherr, Clay Shields, David Wagner, and Wen- chao Zhou. Hidden voice commands. In USENIX Security,

    work page

  7. [7]

    Towards evaluating the robustness of neural networks

    Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In SSP, 2017. 1, 2, 3

    work page 2017

  8. [8]

    Security-oriented picture-in-picture visual modifica- tions

    Thanh-Toan Do, Ewa Kijak, Laurent Amsaleg, and Teddy Furon. Security-oriented picture-in-picture visual modifica- tions. In ICMR, 2012. 2

    work page 2012

Show all 37 references
  1. [9]

    Challenging the security of content-based image re- trieval systems

    Thanh-Toan Do, Ewa Kijak, Teddy Furon, and Laurent Am- saleg. Challenging the security of content-based image re- trieval systems. In MMSP, 2010. 2

    2010

  2. [10]

    Boosting adversarial at- tacks with momentum

    Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial at- tacks with momentum. In CVPR, 2018. 1, 2

    2018

  3. [11]

    Evaluation of GIST descriptors for web-scale image search

    Matthijs Douze, Herv ´e J´egou, Harsimrat Sandhawalia, Lau- rent Amsaleg, and Cordelia Schmid. Evaluation of GIST descriptors for web-scale image search. In CIVR, 2009. 5

    2009

  4. [12]

    Explaining and harnessing adversarial examples

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR,

  5. [13]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR,

  6. [14]

    Ham- ming embedding and weak geometric consistency for large scale image search

    Herve J ´egou, Matthijs Douze, and Cordelia Schmid. Ham- ming embedding and weak geometric consistency for large scale image search. In ECCV, 2008. 5

    2008

  7. [15]

    Cross-dimensional weighting for aggregated deep convolu- tional features

    Yannis Kalantidis, Clayton Mellina, and Simon Osindero. Cross-dimensional weighting for aggregated deep convolu- tional features. In ECCVW, 2016. 3

    2016

  8. [16]

    Adam: A method for stochastic optimization

    Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 2

    2015

  9. [17]

    Imagenet classification with deep convolutional neural net- works

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works. In NeurIPS, 2012. 5

    2012

  10. [18]

    Adver- sarial examples in the physical world

    Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adver- sarial examples in the physical world. In ICLRW, 2017. 2

    2017

  11. [19]

    Universal perturbation attack against image retrieval

    Jie Li, Rongrong Ji, Hong Liu, Xiaopeng Hong, Yue Gao, and Qi Tian. Universal perturbation attack against image retrieval. In arXiv, 2018. 1, 2, 3

    2018

  12. [20]

    Who’s afraid of adversarial queries? the impact of image modifica- tions on content-based image retrieval

    Zhuoran Liu, Zhengyu Zhao, and Martha Larson. Who’s afraid of adversarial queries? the impact of image modifica- tions on content-based image retrieval. In arXiv, 2019. 1, 2

    2019

  13. [21]

    Distinctive image features from scale-invariant keypoints

    David Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 2004. 2

    2004

  14. [22]

    Understanding deep image representations by inverting them

    Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In CVPR,

  15. [23]

    Bags of local convolutional features for scalable instance search

    Eva Mohedano, Kevin McGuinness, Noel E O’Connor, Amaia Salvador, Ferran Marques, and Xavier Giro-i Nieto. Bags of local convolutional features for scalable instance search. In ICMR, 2016. 3

    2016

  16. [24]

    Universal adversarial perturba- tions

    Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturba- tions. In CVPR, 2017. 1, 2

    2017

  17. [25]

    DeepFool: a simple and accurate method to fool deep neural networks

    Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. DeepFool: a simple and accurate method to fool deep neural networks. In CVPR, 2016. 1, 2

    2016

  18. [26]

    Large-scale image retrieval with attentive deep local features

    Hyeonwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, and Bohyung Han. Large-scale image retrieval with attentive deep local features. In ICCV, 2017. 4

    2017

  19. [27]

    Siamese network of deep fisher-vector descriptors for image retrieval

    Eng-Jon Ong, Sameed Husain, and Miroslaw Bober. Siamese network of deep fisher-vector descriptors for image retrieval. In arXiv, 2017. 3

    2017

  20. [28]

    Transferability in machine learning: from phenomena to black-box attacks using adversarial samples

    Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. In arXiv, 2016. 2

    2016

  21. [29]

    Practi- cal black-box attacks against machine learning

    Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practi- cal black-box attacks against machine learning. InASIACCS,

  22. [30]

    Revisiting Oxford and Paris: Large-scale image retrieval benchmarking

    Filip Radenovi ´c, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ond ˇrej Chum. Revisiting Oxford and Paris: Large-scale image retrieval benchmarking. In CVPR, 2018. 5

    2018

  23. [31]

    Fine- tuning CNN image retrieval with no human annotation

    Filip Radenovi ´c, Giorgos Tolias, and Ond ˇrej Chum. Fine- tuning CNN image retrieval with no human annotation. PAMI, 2018. 3

    2018

  24. [32]

    Visual instance retrieval with deep con- volutional networks

    Ali Sharif Razavian, Josephine Sullivan, Stefan Carlsson, and Atsuto Maki. Visual instance retrieval with deep con- volutional networks. ITE Trans. MTA, 2016. 3

    2016

  25. [33]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015. 5

    2015

  26. [34]

    Very deep convo- lutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. InarXiv,

  27. [35]

    In- triguing properties of neural networks

    Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. In- triguing properties of neural networks. In ICLR, 2014. 1, 2, 7

    2014

  28. [36]

    Particular ob- ject retrieval with integral max-pooling of CNN activations

    Giorgos Tolias, Ronan Sicre, and Herv´e J´egou. Particular ob- ject retrieval with integral max-pooling of CNN activations. In ICLR, 2016. 3

    2016

  29. [37]

    Open set adversarial examples

    Zhedong Zheng, Liang Zheng, Zhilan Hu, and Yi Yang. Open set adversarial examples. In arXiv, 2018. 1, 2

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.