Pith. sign in

REVIEW 4 major objections 29 references

LLM-Guided Program Evolution for Targeted Black-Box Attacks on Perceptual Hash Algorithms

T0 review · 4 major / 0 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM-evolved attack programs beat hand-tuned black-box methods at forging perceptual-hash collisions with fewer queries and less distortion.

desk verdict Solid first application of LLM program evolution to black-box PHA second-image attacks; the score gains are real on their fixed evaluator but rest on a 30-pair in-sample design the authors themselves flag. read the letter →

arxiv 2607.11472 v1 pith:YNAFDIIL submitted 2026-07-13 cs.CR

A. Krylov , D. Rakhov , V. Veselova , D. Bolokhov , Oleg Y. Rogov This is my paper · ORCID
classification cs.CR
keywords perceptualhashalgorithmsblack-boxattackstargetedcollisionprogramevolutionLLM-guidedsearchcontentmoderationadversarialrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Perceptual hashes are meant to stay stable under ordinary image edits while still distinguishing different content, so platforms use them to flag known harmful or copyrighted images. This paper asks whether an attacker who can only see the hash distance to a target can still force a source image to collide with that target. The authors show that evolving the attack programs themselves—with language models mutating Python code under a fixed external score that balances success rate, query count, and pixel distortion—finds better collisions than tuning the parameters of existing black-box attacks. On thirty ImageNet pairs and four widely deployed hashes, the best evolved programs cut the composite score by roughly 8–41 percent relative to the best optimized seeds, often with lower L2 distortion and fewer oracle calls. The work therefore treats program search as a practical audit tool for content-moderation hashes that lack formal robustness guarantees.

What carries the argument

The composite score S together with a strict external evaluator: candidate programs may only call a black-box oracle that returns hash distance and enforces a query budget; all success, L2, LPIPS, and S values are recomputed outside the candidate so that bookkeeping cannot inflate results. LLM mutation (via two evolution backends) then searches over full attack programs rather than scalar hyperparameters alone.

What would settle it

Rerun the entire evolution pipeline on a held-out set of image pairs never seen during hyperparameter search or program evolution; if the evolved programs no longer beat the best optimized seeds on the same composite score, the central claim fails.

Watch

Extended reading notes

Core claim

On a controlled 30-pair ImageNet benchmark under a graded black-box distance oracle, LLM-guided evolution of attack programs reduces the composite score S = (L2 + 0.02 Q) / max(ASR, 0.05) relative to the best optimized seed attack by 34.0 percent for pHash, 38.3 percent for PDQ, 8.1 percent for PhotoDNA, and 41.2 percent for NeuralHash, while matching or improving attack success rate and frequently lowering visual distortion or query cost.

Load-bearing premise

Score gains found by searching and ranking on the same fixed thirty image pairs used for tuning, evolution, and final reporting are treated as meaningful evidence of stronger attacks rather than in-sample selection artifacts.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 0 minor

Summary. The paper proposes an LLM-guided program-evolution framework (GigaEvo/OpenEvolve) for black-box targeted second-image attacks on perceptual hash algorithms. Attacks are ranked by a composite score S = (L2 + 0.02 Q) / max(ASR, 0.05) under a graded distance oracle and a strict external evaluator. On four deployed PHAs (pHash, PDQ, PhotoDNA, NeuralHash) and a fixed set of 30 ImageNet pairs, the best evolved programs reduce S relative to the best optimized seed by 34.0%, 38.3%, 8.1%, and 41.2% respectively (Tables 7–8), often with competitive ASR and lower L2 and/or query cost. The authors argue that program-level search is better suited than gradient methods to non-differentiable, quantized hash outputs and that the results motivate stronger robustness evaluation of content-moderation hashes.

Significance. If the score reductions reflect genuine algorithmic improvements rather than in-sample selection, the work is a useful contribution to adversarial evaluation of PHAs: it is fully black-box, handles discrete hash outputs without gradients, and systematically compares hand-written seeds, Optuna-tuned configurations, and LLM-mutated attack programs under a shared external evaluator. The seed portfolio (SimBA, NES, ZO-SignSGD, Prokos, ATKScopes), the two-phase configuration protocol, and the program-level qualitative changes in Table 9 are concrete and reproducible in spirit. Novelty relative to prior hash attacks (Prokos, ATKScopes) and to LLM evolution (AlphaEvolve/OpenEvolve/GigaEvo) is real for this application. The significance is currently limited by the tiny fixed benchmark and single-run evolution design, which the paper itself scopes as controlled benchmark evidence rather than deployment-scale proof of vulnerability.

major comments (4)
  1. §4, §5.3, §6.1 and Tables 7–8: the same 30 ImageNet pairs are used for hyperparameter optimization, program evolution, and final reporting. ASR (Eq. 2), L2, Q and therefore S (Eq. 3) are all computed on this fixed set, so reported score reductions (34.0%/38.3%/8.1%/41.2%) cannot be distinguished from in-sample selection of programs that overfit these particular images and hash-distance landscapes. The Limitations and Conclusion correctly call this “controlled benchmark evidence,” but the Abstract and Introduction present the reductions as demonstrating attack superiority and “previously unreported vulnerabilities.” A held-out pair set (or at least a train/report split with frozen programs) is load-bearing for the central comparative claim.
  2. §5.3: evolution results come from a single run with a single random seed per hash per backend (~1000 candidates), with no cross-seed variance, confidence intervals, or paired significance. Given stochastic LLM mutation and MAP-Elites/island search, a single champion trajectory (Fig. 2) is insufficient to support stable percentage improvements in Table 8. At minimum, multi-seed evolution (or bootstrap over pairs) and reporting of score dispersion are needed before treating the Table 8 deltas as reliable.
  3. §5.3 and Table 8: the comparison of evolved programs to optimized seeds is explicitly not compute-matched (“seed-configuration stage and the evolution stage were not run under a matched total-evaluation budget”). Because evolution evaluates ~1000 full attack programs per hash while Stage 2 only tunes scalar hyperparameters of fixed algorithms, part of the S reduction may be extra search budget rather than the benefit of program-level mutation. A compute-matched seed search (or budget-normalized comparison) is required to isolate the claimed advantage of LLM program evolution.
  4. §5.4 and Table 6 (NeuralHash row): only ATKScopes receives full configuration optimization for NeuralHash; the other four seeds remain at original configurations with ASR ≤ 0.13. The 41.2% improvement in Table 8 is therefore against a partially optimized portfolio, not a fully optimized five-seed baseline. This asymmetry should be reflected in the Abstract’s NeuralHash claim and, ideally, remedied by optimizing at least the next-best viable seeds or by clearly restricting the NeuralHash claim to “vs. optimized ATKScopes.”

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical black-box attack comparison under a pre-defined external score, not a derivation that reduces to its inputs.

full rationale

This is an empirical security/ML paper, not a first-principles derivation. The load-bearing claim is a controlled comparison: LLM-evolved attack programs achieve lower composite score S than per-hash optimized seed attacks on four PHAs (Tables 6–8). S is an author-defined ranking objective (Eq. 3: S = (L2 + 0.02 Q) / max(ASR, 0.05), with a zero-ASR penalty), fixed before evolution and applied by an external evaluator that recomputes metrics from returned images and does not trust attack-reported bookkeeping. Matching thresholds τ0 are fixed operating points (Table 2), not fitted to invent collisions. Reporting that a search method finds lower-S candidates than another under the same pre-defined S is a standard optimization result, not a prediction forced by construction of a free parameter. There is no self-definitional loop (X defined as Y then claimed to derive Y), no uniqueness theorem imported from overlapping authors, no ansatz smuggled via self-citation, and no renaming of a known law as a new derivation. Self-citations (GigaEvo, OpenEvolve, seed attacks) supply tools and baselines, not load-bearing uniqueness. In-sample use of the same 30 ImageNet pairs for tuning, evolution, and reporting is a generalization/validity limitation the paper itself scopes (Limitations, Conclusion); that is not circularity under the stated criteria. Honest finding: derivation chain is self-contained empirical measurement under an external evaluator.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

The central ranking claim rests on a hand-designed composite score, fixed match thresholds, a graded distance oracle, a tiny shared ImageNet pair set, and third-party evolution frameworks—not on a parameter-free robustness theorem. Invented machinery is the evaluation/score protocol and the evolved attack programs themselves; free knobs in S and budgets directly shape who ‘wins’.

free parameters (5)
  • Composite score query weight 0.02 (=1/50)
    Sets the trade-off between L2 and oracle queries in S; chosen by authors, not derived from first principles.
  • ASR floor 0.05 and zero-success penalty 10000
    Denominator and additive penalty in Eqs. 3–4 control ranking of low/zero ASR runs; ad hoc design choices that affect selection.
  • Match thresholds τ0 (12/92/3855/17)
    Fixed per-hash operating points (Table 2) that define success; not optimized but load-bearing for ASR counts.
  • Per-hash query budgets (10k/10k/20k/30k)
    Hard resource limits that shape feasible ASR and S; chosen protocol parameters.
  • Evolution budget (~1000 candidates/hash/backend) and single random seed
    Search effort and stochasticity of reported champions; not averaged.
assumptions (5)
  • domain assumption A graded black-box oracle returning only hash distance (not hard labels, gradients, or target image) is the correct threat model for the claimed attacks.
    Stated in §3; hard-label and rate-limited APIs are out of scope.
  • domain assumption Hamming (or L1 for PhotoDNA) distance below fixed τ0 is the right success criterion for a second-image collision.
    Eq. 1 and Table 2; industrial decision logic may differ.
  • ad hoc to paper Thirty resized ImageNet pairs are an adequate controlled benchmark for comparing seed vs evolved programs.
    §4; same split used for optimization, evolution, and reporting.
  • domain assumption External recomputation of metrics is authoritative over attack-reported bookkeeping.
    §3 evaluator contract; standard for untrusted generated code.
  • domain assumption LLM mutation plus MAP-Elites/lineage search can produce valid run_attack programs under the shared interface.
    §2.3–§3; relies on cited GigaEvo/OpenEvolve behavior.
invented entities (2)
  • Composite attack score S
    purpose: Single scalar for configuration optimization and program evolution ranking.
    Defined in Eqs. 3–4; not a standard PHA metric; LPIPS deliberately excluded from S.
  • EvoHash-style external evaluator + seed portfolio pipeline
    purpose: Unify baselines, Optuna tuning, and LLM program evolution under one contract.
    Framework contribution of the paper; value is empirical ranking, not an external physical entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM-Guided Program Evolution for Targeted Black-Box Attacks on Perceptual Hash Algorithms." pith.science (2026). https://pith.science/paper/YNAFDIIL

@misc{pith2026260711472,
  author       = {Pith},
  title        = {Pith review of: LLM-Guided Program Evolution for Targeted Black-Box Attacks on Perceptual Hash Algorithms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YNAFDIIL}},
  note         = {Machine review of arXiv:2607.11472}
}
read the original abstract

Perceptual hash algorithms (PHAs) are widely deployed to detect image forgery under benign transformations, yet their robustness against adversarially chosen perturbations remains poorly understood and rarely comes with provable guarantees. We propose a novel evolutionary framework based on GigaEvo and OpenEvolve for targeted second-image attacks on perceptual hash algorithms. We assess attack performance using a composite score that jointly accounts for the fraction of adversarial images whose normalized Hamming distance to the target hash falls below threshold p (Attack Success Rate), the number of queries issued to the hash function, and the L2 distortion relative to the original image. Experiments on four deployed PHAs (pHash, PDQ, PhotoDNA, NeuralHash) across 30 ImageNet image pairs demonstrate that our evolutionary approach achieves comparable or better ASR than existing black-box baselines using substantially fewer queries to the hash function, while simultaneously producing adversarial images with lower L2 distortion relative to the originals. The best evolved programs reduce the pre-defined composite attack score relative to the best optimized seed by 41.2% for NeuralHash, 38.3% for PDQ, 34.0% for pHash, and 8.1% for PhotoDNA. Unlike gradient-based methods, our framework requires no internal knowledge of PHA architectures and naturally handles the non-differentiable, discretized nature of hash outputs. These results reveal previously unreported vulnerabilities in widely deployed content-moderation pipelines and motivate the development of provably robust perceptual hashing 1schemes.

Figures

Figures reproduced from arXiv: 2607.11472 by the authors.

Figure 1
Figure 1. Proposed framework. Per-hash tuned seed attacks seed LLM-guided program evolution against the black-box oracle O returning only the hash distance d. A single external evaluator scores each candidate with S (lower is better). 2.3 LLM–based evolution Evolutionary computation has a well–established tradition in combinatorial opti￾mization. Its integration with large language models represents an emerging and rapidly ad… view at source ↗
Figure 2
Figure 2. Normalized OpenEvolve champion score during program evolution. Lower val￾ues indicate improvement over the initial seed program. 7 Discussion Program evolution improves beyond seed tuning. Configuration optimiza￾tion substantially improves pHash, PDQ, and NeuralHash, but evolved programs still reduce the final score for every hash. This indicates that the useful search space includes algorithmic changes rather than … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 2 canonical work pages

  1. [1]

    Akiba, T., Sano, S., Yanase, T., Ohta, T., Koyama, M.: Optuna: A next-generation hyperparameter optimization framework (2019), https://arxiv.org/abs/1907.10902

    arXiv 2019

  2. [2]

    In: 2022 IEEE International Conference on Cyber Security and Resilience (CSR)

    Alkhowaiter, M., Almubarak, K., Zou, C.: Evaluating perceptual hashing algo- rithms in detecting image manipulation over social media platforms. In: 2022 IEEE International Conference on Cyber Security and Resilience (CSR). pp. 149–156 (2022). https://doi.org/10.1109/CSR54599.2022.9850288

    work page doi:10.1109/csr54599.2022.9850288 2022

  3. [3]

    The Guardian22(2013)

    Arthur, C.: Twitter to introduce photodna system to block child abuse images. The Guardian22(2013)

    2013

  4. [4]

    PetaPixel, August (2014)

    Burgett, G.: Photodna lets google, fb and others hunt down child pornography without looking at your photos. PetaPixel, August (2014)

    2014

  5. [5]

    SocArXiv (2021)

    Cobbe, J.: Data protection, eprivacy, and the prospects for apple’s on-device csam detection system in europe. SocArXiv (2021)

    2021

  6. [6]

    Facebook Newsroom (2019)

    Davis, A., Rosen, G.: Open-sourcing photo-and video-matching technology to make the internet safer. Facebook Newsroom (2019)

    2019

  7. [7]

    In: 2009 IEEE Conference on Computer Vision and Pattern Recognition

    Deng, J., et al.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. pp. 248–255 (2009). https://doi.org/10.1109/CVPR.2009.5206848

    doi:10.1109/cvpr.2009.5206848 2009

  8. [8]

    ArXivabs/2011.09473(2020), https://api.semanticscholar.org/CorpusID:227013585

    Dolhansky, B., Canton-Ferrer, C.: Adversarial collision attacks on image hashing functions. ArXivabs/2011.09473(2020), https://api.semanticscholar.org/CorpusID:227013585

    arXiv 2011

Show all 29 references
  1. [9]

    Journal of Online Trust and Safety 1(1) (2021)

    Farid, H.: An overview of perceptual hashing. Journal of Online Trust and Safety 1(1) (2021)

    2021

  2. [10]

    Fishkov, A., Schweighofer, K., Ielanskyi, M., Kotelevskii, N., Guizani, M., Panov, M.: Uncertainty quantification for regression using proper scoring rules (2025), https://arxiv.org/abs/2509.26610

    2025

  3. [11]

    Guo,C.,Gardner,J.R.,You,Y.,Wilson,A.G.,Weinberger,K.Q.:Simpleblack-box adversarial attacks (2019), https://arxiv.org/abs/1905.07121

    2019 arXiv

  4. [12]

    In: Rudolph, S., Dau, F., Kuznetsov, S.O

    Ignatov, D.I., Kuznetsov, S.O.: Frequent itemset mining for clustering near dupli- cate web documents. In: Rudolph, S., Dau, F., Kuznetsov, S.O. (eds.) Conceptual Structures: Leveraging Semantic Technologies. pp. 185–200. Springer Berlin Hei- delberg, Berlin, Heidelberg (2009)

    2009

  5. [13]

    Ilyas, A., Engstrom, L., Athalye, A., Lin, J.: Black-box adversarial attacks with limited queries and information (2018), https://arxiv.org/abs/1804.08598

    2018 arXiv

  6. [14]

    Retrieved from Microsoft News Center: https://news

    Ith, T.: Microsoft’s photodna: Protecting children and businesses in the cloud. Retrieved from Microsoft News Center: https://news. microsoft. com/features/microsofts-photodna-protecting-children-and-businesses-in-the- cloud (2015) LLM-Guided Program Evolution for Targeted Bla...

    2015

  7. [15]

    Khrulkov, V., et al.: Gigaevo: An open source optimization framework powered by llms and evolution algorithms (2025), https://arxiv.org/abs/2511.17592

    2025

  8. [16]

    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization (2017), https://arxiv.org/abs/1412.6980

    2017 arXiv

  9. [17]

    Klinger, E., Starkweather, D.: phash: The open source perceptual hash library (2013)

    2013

  10. [18]

    Cryptology ePrint Archive, Paper 2024/1869 (2024), https://eprint.iacr.org/2024/1869

    Leblanc-Albarel, D., Preneel, B.: Black-box collision attacks on widely deployed perceptual hash functions. Cryptology ePrint Archive, Paper 2024/1869 (2024), https://eprint.iacr.org/2024/1869

    2024

  11. [19]

    In: International Conference on Learning Representations (2019), https://openreview.net/forum?id=BJe-DsC5Fm

    Liu, S., Chen, P.Y., Chen, X., Hong, M.: signSGD via zeroth-order oracle. In: International Conference on Learning Representations (2019), https://openreview.net/forum?id=BJe-DsC5Fm

    2019

  12. [20]

    Madden, J., Bhavsar, M., Dorje, L., Li, X.: Robustness of practical percep- tual hashing algorithms to hash-evasion and hash-inversion attacks (2024), https://arxiv.org/abs/2406.00918

    2024 arXiv

  13. [21]

    Novikov, A., et al.: Alphaevolve: A coding agent for scientific and algorithmic discovery (2025), https://arxiv.org/abs/2506.13131

    2025 arXiv

  14. [22]

    In: 32nd USENIX Security Symposium (USENIX Security 23)

    Prokos, J., et al.: Squint hard enough: Attacking perceptual hashing with ad- versarial machine learning. In: 32nd USENIX Security Symposium (USENIX Security 23). pp. 211–228. USENIX Association, Anaheim, CA (Aug 2023), https://www.usenix.org/conference/usenixsecurity23/presen...

    2023

  15. [23]

    In: T.Y.S.S., S., Rodriguez, J.D., de Gibert, O

    Seleznyov, M., Korbut, D., Moskvoretskii, V., Somov, O., Panchenko, A., Tu- tubalina, E.: Evolutionary search for automated design of uncertainty quantifi- cation methods. In: T.Y.S.S., S., Rodriguez, J.D., de Gibert, O. (eds.) Proceed- ings of the 64th Annual Meeting of the A...

    2026 doi

  16. [24]

    Sharma, A.: Openevolve: an open-source evolutionary coding agent (2025), https://github.com/algorithmicsuperintelligence/openevolve

    2025

  17. [25]

    In: 2022 ACM Conference on Fairness Accountability and Transparency

    Struppek, L., Hintersdorf, D., Neider, D., Kersting, K.: Learning to break deep perceptual hashing: The use case neuralhash. In: 2022 ACM Conference on Fairness Accountability and Transparency. p. 58–69. FAccT ’22, ACM (Jun 2022). https://doi.org/10.1145/3531146.3533073, http:...

    2022 doi

  18. [26]

    Watanabe, S.: Tree-structured parzen estimator: Understanding its algo- rithm components and their roles for better empirical performance (2025), https://arxiv.org/abs/2304.11127

    2025 arXiv

  19. [27]

    Zauner, C.: Implementation and benchmarking of perceptual image hash functions (2010), https://api.semanticscholar.org/CorpusID:17075066

    2010

  20. [28]

    Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The un- reasonable effectiveness of deep features as a perceptual metric (2018), https://arxiv.org/abs/1801.03924

    2018 arXiv

  21. [29]

    In: 34th USENIX Security Symposium (USENIX Security 25)

    Zhang, Y., et al.: Atkscopes: Multiresolution adversarial perturbation as a unified attack on perceptual hashing and beyond. In: 34th USENIX Security Symposium (USENIX Security 25). pp. 5913–5930. USENIX Association, Seattle, WA (Aug 2025), https://www.usenix.org/conference/us...

    2025

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.