GEO-Bench: Benchmarking Ranking Manipulation in Generative Engine Optimization

Gengpei Qi; Ojas Nimase; Xiyang Hu; Yue Zhao; Zhe Chen

REVIEW 1 major objections 2 minor 1 cited by

GEO-Bench shows black-box rewriting matches gradient attacks at promoting LLM rankings while evading detection.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-29 11:11 UTC pith:BYQNBQIL

load-bearing objection GEO-Bench standardizes GEO attack comparisons under one protocol and ranker, which is useful, but the single fixed Llama-3.1-8B ranker and five datasets make the headline claims about black-box superiority and evasion hard to generalize. the 1 major comments →

arxiv 2605.29107 v2 pith:BYQNBQIL submitted 2026-05-27 cs.CR cs.AI

GEO-Bench: Benchmarking Ranking Manipulation in Generative Engine Optimization

Ojas Nimase , Zhe Chen , Gengpei Qi , Yue Zhao , Xiyang Hu This is my paper

classification cs.CR cs.AI

keywords generative engine optimizationranking manipulationbenchmarkblack-box attacksgradient attacksLLM rankingdetection evasion

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces GEO-Bench to compare ranking manipulation methods for large language models under a single consistent protocol. It evaluates black-box prompt attacks, white-box gradient attacks, and white-hat strategies across five datasets using one fixed ranker and shared metrics for rank promotion success and text naturalness. The results establish that black-box content rewriting achieves comparable or better promotion outcomes than gradient methods, generates more fluent text, and evades keyword and perplexity detectors on some domains, while access level does not determine overall strength. This standardization makes direct comparisons possible for the first time.

Core claim

GEO-Bench demonstrates that black-box content rewriting matches or exceeds gradient-based attacks on rank promotion while producing more fluent text and can evade both keyword- and perplexity-based detection on some domains. It further shows that the access model does not predict attack strength. By unifying datasets, attack implementations, and metrics for effectiveness and stealth, the benchmark enables the first direct comparison across adversarial attack paradigms.

What carries the argument

GEO-Bench benchmark, which applies fixed datasets, a single Llama-3.1-8B-Instruct ranker, and metrics for effectiveness (NRG, Success@α, Promote@α) and stealth (keyword violation rate, perplexity ratio) to black-box, white-box, and white-hat methods.

Load-bearing premise

That the five chosen datasets together with the single fixed ranker and listed metrics are sufficient to draw general conclusions about relative attack strength and detectability across real generative engines.

What would settle it

An experiment on a new dataset or different ranker in which gradient-based attacks show clearly higher promotion rates and lower detection evasion than black-box rewriting methods.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

GEO-Bench standardizes GEO attack comparisons under one protocol and ranker, which is useful, but the single fixed Llama-3.1-8B ranker and five datasets make the headline claims about black-box superiority and evasion hard to generalize.

read the letter

The paper's main contribution is GEO-Bench, which puts several GEO attack families—TAP, Zero-Shot, STS, RAF, StealthRank, and C-SEO—on the same datasets, the same Llama-3.1-8B-Instruct ranker, and the same metrics for promotion and stealth. That unification is the real step forward; prior work evaluated these methods separately, so direct comparisons were missing.

It does a clean job showing the effectiveness-stealth tradeoff and that black-box rewriting can match or beat gradient-based methods on rank promotion while looking more fluent and sometimes evading the two detection checks they test. The finding that access model does not predict attack strength is also worth noting.

The soft spot is exactly what the stress-test flags. All comparisons run against one open-weight ranker and five datasets. Real generative engines vary in architecture, fine-tuning, and query handling, so the ordering of attack strength and the evasion results could shift under different rankers. The abstract does not mention error bars, multiple random seeds, or sensitivity checks on the ranker choice, which leaves the central claims resting on a narrow base.

This is the kind of benchmark paper that belongs in the literature because it gives the community a common yardstick. Researchers building attacks or defenses in LLM ranking will get value from the standardized setup, even if they will want to re-run pieces on their own models. It deserves a serious referee rather than a desk reject; the unification is solid enough to justify review time, with the main revision likely being more robustness tests across rankers.

Referee Report

1 major / 2 minor

Summary. The paper introduces GEO-Bench, a standardized benchmark for comparing generative engine optimization (GEO) ranking-manipulation attacks. It unifies black-box prompt-based methods (TAP, Zero-Shot), white-box gradient-based attacks (STS, RAF, StealthRank), and ten C-SEO strategies; evaluates them on five datasets against a single fixed Llama-3.1-8B-Instruct ranker; and reports effectiveness via NRG, Success@α, Promote@α and stealth via keyword violation rate and perplexity ratio. The headline findings are that black-box rewriting matches or exceeds gradient-based attacks on promotion while producing more fluent text and evading keyword- and perplexity-based detectors on some domains, and that the access model does not predict attack strength.

Significance. If the comparative ordering proves robust, the benchmark supplies the first apples-to-apples protocol for GEO attacks and thereby supports systematic development of detection methods. The explicit unification of previously incomparable attack families is a clear contribution to the cs.CR literature on LLM ranking manipulation.

major comments (1)

[Evaluation protocol and main results] The central claims—that black-box rewriting matches or exceeds gradient-based attacks and that access model does not predict strength—rest entirely on comparisons performed with one fixed open-weight ranker (Llama-3.1-8B-Instruct) and the five listed datasets. Because the paper fixes the ranker and does not report results across alternative architectures, fine-tunes, or commercial engines, any model-specific bias in how perturbations affect ranking directly determines the reported ordering of attack paradigms. This is load-bearing for the headline conclusions.

minor comments (2)

The abstract states comparative results yet the manuscript provides no methods section, no error bars, no dataset statistics, and no code or data release; these omissions prevent independent verification of the reported metrics.
The five datasets and single ranker choice should be justified with an explicit limitations paragraph rather than left implicit.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback on the evaluation protocol. We address the major comment below and outline revisions to clarify the scope of our claims while preserving the benchmark's contribution as a standardized, reproducible protocol.

read point-by-point responses

Referee: [Evaluation protocol and main results] The central claims—that black-box rewriting matches or exceeds gradient-based attacks and that access model does not predict strength—rest entirely on comparisons performed with one fixed open-weight ranker (Llama-3.1-8B-Instruct) and the five listed datasets. Because the paper fixes the ranker and does not report results across alternative architectures, fine-tunes, or commercial engines, any model-specific bias in how perturbations affect ranking directly determines the reported ordering of attack paradigms. This is load-bearing for the headline conclusions.

Authors: We agree that the headline findings are obtained under a single fixed ranker and that this design choice makes the reported ordering specific to Llama-3.1-8B-Instruct. The benchmark deliberately fixes the ranker, datasets, and metrics to enable the first controlled, apples-to-apples comparison across previously incomparable attack families; varying the ranker would confound the comparison of attack methods themselves. Nevertheless, the referee correctly identifies that the claims would benefit from explicit qualification. In the revised manuscript we will (1) add a dedicated Limitations subsection that states the results are tied to the chosen open-weight model and may not generalize to commercial engines or fine-tuned variants, (2) qualify the abstract and conclusion sentences to read “under the Llama-3.1-8B-Instruct ranker” rather than implying universality, and (3) include a short additional experiment on a second open-weight model (e.g., Mistral-7B-Instruct) to illustrate sensitivity, time permitting. We believe these changes address the load-bearing concern without altering the core contribution of the unified protocol. revision: yes

Circularity Check

0 steps flagged

Empirical benchmark paper with no derivations or self-referential reductions

full rationale

The paper presents GEO-Bench as a standardized empirical evaluation of existing GEO attacks (TAP, Zero-Shot, STS, RAF, StealthRank, C-SEO) on five datasets using one fixed external ranker (Llama-3.1-8B-Instruct) and listed metrics (NRG, Success@α, Promote@α, keyword violation rate, perplexity ratio). No equations, fitted parameters, or derivations appear in the provided text; the central claims are direct empirical observations from running the attacks under a common protocol. No self-citation chains, ansatzes, or renamings reduce any result to the paper's own inputs by construction. The fixed ranker and datasets are external choices, not self-defined quantities.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The evaluation rests on the assumption that the chosen ranker and metrics generalize; no free parameters or invented entities are introduced in the abstract.

axioms (1)

domain assumption Llama-3.1-8B-Instruct serves as a representative fixed ranker for generative engines.
Used as the sole evaluation model in all experiments.

pith-pipeline@v0.9.1-grok · 5783 in / 1181 out tokens · 37785 ms · 2026-06-29T11:11:38.162691+00:00 · methodology

0 comments

read the original abstract

Large language models (LLMs) increasingly rank products, documents, and recommendations for user queries, which makes manipulating these rankings a growing concern for fairness and information integrity. Research on generative engine optimization (GEO) has produced many manipulation methods, but each is evaluated on its own dataset with its own metrics, so their relative strength and detectability stay unclear. We present GEO-Bench, a benchmark that evaluates GEO ranking-manipulation attacks under one protocol. It unifies black-box prompt-based attacks (TAP, Zero-Shot), white-box gradient-based attacks (STS, RAF, StealthRank), and ten white-hat C-SEO strategies. We score every method on five datasets against a fixed open-weight ranker (Llama-3.1-8B-Instruct), using metrics for both effectiveness (NRG, Success@{\alpha}, Promote@{\alpha}) and stealth (keyword violation rate, perplexity ratio). Our evaluation shows that effectiveness and stealth trade off across adversarial attacks, that black-box content rewriting matches or exceeds gradient-based attacks on rank promotion while producing more fluent text and can evade both keyword- and perplexity-based detection on some domains, and that the access model does not predict attack strength. By standardizing datasets, attack implementations, and metrics, GEO-Bench enables the first direct comparison across these attack paradigms and supports the development of detection methods.

Figures

Figures reproduced from arXiv: 2605.29107 by Gengpei Qi, Ojas Nimase, Xiyang Hu, Yue Zhao, Zhe Chen.

**Figure 1.** Figure 1: Effectiveness–stealth tradeoff. No adversarial attack is at once effective, keyword-stealthy, and fluent; only white-hat rewriting (the star) escapes the tradeoff. Each marker is a method’s mean over the five datasets: effectiveness (NRG, higher is stronger promotion) against keyword violation rate (KVR, lower is stealthier), with marker size the mean perplexity ratio (larger = less fluent) and color the m… view at source ↗

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Optimizing Visibility in Generative Engines: A Critical Survey of Generative Engine Optimization (2023-2026)
cs.IR 2026-07 conditional novelty 5.0

A critical review of GEO research concludes that already-retrieved content can improve citation and use, but no tested technique reliably raises organic discoverability or downstream traffic across engines.

Reference graph

Works this paper leans on

2 extracted references · 1 canonical work pages · cited by 1 Pith paper

[1]

Shopping queries dataset: A large-scale esci benchmark for improving product search.arXiv preprint arXiv:2206.06588,

Shopping queries dataset: A large-scale esci benchmark for improving product search.Preprint, arXiv:2206.06588. Yiming Tang, Yi Fan, Chenxiao Yu, Tiankai Yang, Yue Zhao, and Xiyang Hu. 2025. Stealthrank: Llm rank- ing manipulation via stealthy prompt optimization. Preprint, arXiv:2504.05804. Tiancheng Xing, Jerry Li, Yixuan Du, and Xiyang Hu

work page arXiv 2025
[2]

high final rank

Are llms reliable rankers? rank manipulation via two-stage token optimization. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), San Diego, California, United States. Association for Computational Linguistics. A Dataset Construction and Processing All datasets are standardized into a unified...

2024

[1] [1]

Shopping queries dataset: A large-scale esci benchmark for improving product search.arXiv preprint arXiv:2206.06588,

Shopping queries dataset: A large-scale esci benchmark for improving product search.Preprint, arXiv:2206.06588. Yiming Tang, Yi Fan, Chenxiao Yu, Tiankai Yang, Yue Zhao, and Xiyang Hu. 2025. Stealthrank: Llm rank- ing manipulation via stealthy prompt optimization. Preprint, arXiv:2504.05804. Tiancheng Xing, Jerry Li, Yixuan Du, and Xiyang Hu

work page arXiv 2025

[2] [2]

high final rank

Are llms reliable rankers? rank manipulation via two-stage token optimization. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), San Diego, California, United States. Association for Computational Linguistics. A Dataset Construction and Processing All datasets are standardized into a unified...

2024