Exploring LLM biases to manipulate AI search overview

Roman Smirnov

LLM search overviews pick sources by relative bias, and a length-limited snippet rewriter can game that choice.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 16:28 UTC

load-bearing objection Solid constrained GEO-style RL study on real-length snippets; the comparative-advantage finding is the real takeaway, but train-on-judge and a simplified ID selector keep the production claims provisional. the 5 major comments →

arxiv 2605.00012 v1 submitted 2026-03-30 cs.IR cs.AIcs.CL

Exploring LLM biases to manipulate AI search overview

Roman Smirnov This is my paper

classification cs.IR cs.AIcs.CL

keywords LLM Overviewsearch result biassnippet rewritingreinforcement learningcomparative advantagecontext poisoninggenerative engine optimizationAI search

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Search systems that ask a large language model to pick a few sources and write an overview of the results inherit the model’s content preferences. This paper shows those preferences are stable under reordering, prompt tweaks, and model swaps, then trains a small rewriter that only edits the short search-engine snippet so the rewritten page becomes one of the three sources the overview cites. The rewriter must keep roughly the original length and sees the other candidates, so it is learning comparative advantage rather than a universal “good snippet” style. On held-out shopping queries the rewrite raises citation rates across several randomizations. The same setup can also pull poisoned wording into a rewritten snippet when a reference is contaminated, so the overview may cite inaccurate or harmful text. The practical stakes are high: overviews already cut publisher clicks, so whoever can steer citation can steer attention and, in the worst case, inject bad information into the answer users actually read.

Core claim

LLM Overview systems exhibit stable selection biases, and under a restrictive snippet-only policy with length limits, reinforcement learning can rewrite snippets so they become more likely to be among the three sources the overview cites; those selections are driven by comparative rather than absolute advantages among candidates, and context poisoning through reference snippets can produce inaccurate or harmful cited content.

What carries the argument

Reference-conditioned snippet rewriting under Dr.GRPO: a small LoRA policy that rewrites only the target snippet while seeing the other candidates, rewarded by a weighted mix of soft length match and whether the overview judge cites the rewritten result.

Load-bearing premise

That a simplified judge which only returns three IDs from URL-title-snippet dictionaries, mostly using a nano model, is close enough to real multi-stage overview systems that beating it counts as manipulation of production search overviews.

What would settle it

Run the same trained rewriter against a production-style overview pipeline that visits full pages or uses a model whose selection is dominated by URL rather than snippet content, and check whether the rewritten snippets still raise citation rates on the same held-out queries.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid constrained GEO-style RL study on real-length snippets; the comparative-advantage finding is the real takeaway, but train-on-judge and a simplified ID selector keep the production claims provisional.

read the letter

Roman’s paper is a practical systems note on whether you can push LLM Overview citation by rewriting only the short snippets a search engine actually returns. The useful deltas versus AutoGEO and Rewrite-to-Rank are the restrictive setup: real engine-length snippets, length soft-limit in the reward (DAPO-style), and reference-conditioned rewriting so the policy sees the competing set. Unconditional rewrite failed; conditional worked. That is the cleanest result—selection looks comparative, not absolute—and it matches the intuition that “best in this shortlist” is different from “good in isolation.”

Bias evidence is directionally fine: 90 shopping queries, binomial null under random selection of 3/N, and a concrete Starbucks table that also shows the earned-media pattern. RL training curves and a 90-sample held-out test with order/prompt/model randomizations show the policy lifts citation share for several GPT-4.1/5-nano-class judges; gpt-5-mini is an honest failure case they attribute to URL bias. The reference-context poisoning path (poison a non-target snippet, rewrite the target, get harmful text cited) is a real safety flag even if the attack suite is small and self-described as limited.

Soft spots are real but not hidden. Main reward and primary eval use the same simplified “return three IDs” judge (mostly gpt-4.1-nano). They claim preliminary correlation with fuller prompts and budget reasons; that is plausible for a lab study and not pure circularity once they add randomizations and a model-change condition, but it is still a surrogate. No CIs, no significance tests, shopping-only queries, no code/data. Strong “prove” language overreaches what a closed-loop ID selector can show about multi-stage production overviews.

Who it is for: people doing GEO, ranking integrity, or LLM-search safety who want a constrained, reproducible-style attack surface rather than invisible-instruction SEO. Worth a serious referee if the venue cares about empirical IR/safety systems work; expect pressure to quantify transfer off the ID judge and to release artifacts. I would engage the comparative-advantage and length-constrained RL pieces; I would not treat the production-manipulation claim as settled.

Referee Report

5 major / 0 minor

Summary. The paper argues that LLM Overview systems (LLM-based selection and summarization of search results) exhibit stable content-selection biases, demonstrated via permutation/randomization experiments on ~90 Amazon shopping queries under a binomial null of unbiased citation (p=3/N). It then trains a small policy (gemma-3-1b-it LoRA, Dr.GRPO/DAPO-style RL) that rewrites only search-engine snippets, conditioned on competing snippets and under length soft-limits, to raise the probability that a target result is among the three sources cited by a simplified overview judge (mostly gpt-4.1-nano returning IDs). Success is reported on a held-out 90-sample set across randomizations; the authors conclude that preferences are comparative rather than absolute and that reference-context poisoning can inject inaccurate or harmful content into cited snippets.

Significance. If the core results transfer beyond the surrogate, the work is significant for generative-engine optimization, search integrity, and safety of AI overviews: it shows that realistic constraints (snippet-only edits, length limits, no full-page access) still allow RL-driven preference manipulation, and it surfaces a comparative-advantage mechanism plus a concrete context-poisoning vector. Strengths include the deliberately restrictive experimental design that mirrors web-search constraints, the explicit length-reward component to block a known reward-hack, the comparative-vs-absolute insight, and the safety probes in §7. These are concrete, falsifiable contributions relative to concurrent rewrite-to-rank / AutoGEO-style work that often operates on longer page text with fewer length or observability limits.

major comments (5)

§6.1 and §6.5 / Figure 4: The primary reward and the main evaluation both use the same simplified overview judge (prompted to return three IDs from URL/title/snippet dicts, predominantly gpt-4.1-nano). This is classic train-on-the-metric circularity. The model-change randomization (GPT-5-nano) and order/URL shuffles only partially break the loop; no held-out correlation is reported between the ID-selector and full multi-stage overview pipelines (query rewriting, page reading, joint selection+generation) that the paper itself diagrams in Figure 1. Without that correlation, the claim that RL manipulates production LLM Overviews does not follow from success against the surrogate.
§5 binomial model and Table 1: The null assumes every one of N results is equally relevant and that the K randomizations are independent Bernoulli trials with p=3/N. The shopping-query domain and the earned-media pattern in Table 1 make equal relevance implausible; the paper asserts relevance but supplies no independent relevance labels or inter-annotator check. Consequently the reported 48/90 full-persistence and 30/90 never-selected cases cannot be cleanly attributed to model bias versus residual relevance ranking. A load-bearing claim (“LLM Overview is biased”) rests on this untested null.
§6.5 evaluation and Figure 4: Test-set results are given only as average shares across randomizations with no confidence intervals, no per-query paired significance tests, and no baseline comparison against non-RL rewriters (e.g., zero-shot or SFT-only). The claim that “reinforcement learning in most of the cases can optimize snippet’s content” therefore lacks a quantitative statistical footing on the 90-sample test set.
§7 Attacks: The only successful attack (reference poisoning) is illustrated with a single query and a single generated snippet; the authors themselves state the experiments “are not broad enough” and were run on “a very limited dataset.” The safety claim that “context poisoning attacks can lead to inaccurate or harmful results” is therefore anecdotal and cannot support the abstract-level assertion without a larger, quantified attack success rate and a clear definition of “harmful.”
§8 and the gpt-5-mini failure: The policy fails to improve reward against gpt-5-mini, which the authors attribute to URL bias. This is an important negative result, yet it is left as an assumption. Because the abstract claims success “in most of the cases,” the manuscript needs either (a) a systematic characterization of which overview models are vulnerable or (b) an explicit scope restriction so that the positive claims are not over-generalized.

Circularity Check

1 steps flagged

Mild train-on-surrogate-judge loop only; no definitional reduction of the comparative-advantage or bias claims.

specific steps

fitted input called prediction [§6.1 Reward function; §6.5 Results / Figure 4]
"We use LLM Overview generator's output as the main component of the reward function. Taking the generated summary, we evaluated whether the rewritten snippet is cited inside the overview. ... To test the results of the training, we took the resulting policy and tested it on the test dataset. ... We used the same LLM Overview system we used as a reward during the reinforcement learning training, however, we additionally applied randomization (permutation) techniques"

The policy parameters are fitted to maximize citation probability under the surrogate judge; the headline success metric is then the same citation probability (plus randomizations) under that judge. On the surrogate the lift is therefore the optimized objective rather than an independent prediction. The paper mitigates this with held-out queries and model-change, so the reduction is only partial.

full rationale

This is an empirical RL paper, not a first-principles derivation. The sole mild circularity is that the policy is optimized against an LLM Overview ID-selector (mostly gpt-4.1-nano) and then primarily scored by the same class of selector (Section 6.5, Figure 4), so reported lift is the training objective on the surrogate. Randomizations, a model-change condition (GPT-5-nano), and a held-out query set partially break the loop; the paper itself flags the surrogate as a simplification justified by preliminary correlation (Section 6.1). The bias-persistence statistics (Section 5, binomial null) and the comparative-vs-absolute insight (unconditional rewrite failed, reference-conditioned succeeded) are independent experimental observations, not forced by definition or self-citation. No uniqueness theorems, no ansatz smuggling, no load-bearing self-citations, and no renaming of known results. Score 2 reflects only the standard train-eval metric overlap, which does not collapse the central claims by construction.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 1 invented entities

The paper is empirical systems work, not a formal derivation. Load-bearing content is mostly domain assumptions about how LLM Overview pipelines work, a simplified selection surrogate, and many training/reward hyperparameters chosen by hand. No new physical entities; ‘LLM Overview’ names existing product class. Free parameters are the usual RL and reward knobs that the success claim depends on.

free parameters (6)

length_reward_weight = 0.2
Main successful run uses a weighted sum with length reward weight 0.2 against LLM Overview reward; other weightings failed or were less robust (§6.4).
GRPO_sampling_temperature = 3.0
Policy sampling temperature set to 3.0 for exploration during GRPO (§6.3); not derived.
candidates_per_sample = 8
Number of GRPO candidates per sample fixed at 8 (§6.3).
base_learning_rate = 1e-5
LoRA training LR set to 1e-5 (§6.3).
urls_k_cited = 3
Overview always asked for exactly 3 cited IDs; shapes both reward sparsity and binomial null p=3/N (§5–6).
similarity_embedder_and_optional_weight = dropped / optional
e5-small cosine similarity considered then dropped for speed; when used it slowed reward growth (§6.1, §6.4).

axioms (5)

domain assumption A simplified ID-selection overview prompt on URL/title/snippet dicts correlates strongly enough with real LLM Overview selection/generation to support manipulation claims.
Explicitly justified in §6.1 by preliminary correlation experiments and cost control; central to interpreting RL reward as overview manipulation.
ad hoc to paper Under a null of unbiased selection among N relevant results, each snippet is cited with p=3/N independently across K randomizations (binomial model).
Used in §5 to call persistence ‘bias’; independence and equal relevance are strong idealizations for real search result sets.
domain assumption Search-engine snippets have strict length limits and the policy may only edit snippets, not titles/URLs/pages—matching realistic web constraints.
Design premise of the experimental setup (abstract, §6); motivates length soft-limiting after DAPO.
ad hoc to paper Rewriting snippets to be preferred by an overview model is largely within the base small LM’s capability distribution, so SFT warmup can be dropped.
Stated in §6.4; if false, the no-SFT regime understates required training.
standard math Standard RL fine-tuning machinery (LoRA on gemma-3-1b-it, Dr.GRPO without KL, DAPO-style length shaping) is an appropriate optimizer for this preference.
Methods imported from cited GRPO/Dr.GRPO/DAPO/QLoRA literature (§6.3).

invented entities (1)

LLM Overview (as experimental system class) independent evidence
purpose: Name the product pattern of LLM selection+summary over search results that the bias and attack claims target.
Not a new physical entity; operational label for GPT-search / AI Overview-like pipelines. No independent ontology beyond existing products.

pith-pipeline@v1.1.0-grok45 · 14754 in / 3886 out tokens · 44410 ms · 2026-07-13T16:28:58.391323+00:00 · methodology

0 comments

read the original abstract

Modern large language models (LLMs) are used in many business applications in general, and specifically in web search systems and applications that generate overviews of search results - LLM Overview systems. Such systems are using an LLM to select most relevant sources from search results and generate an answer to the user's query. It is known from many studies that LLMs have different biases, in LLM Overview application both the source selection and answer generation stages may be affected by the biases of LLMs (here we are focusing mainly on the selection stage). This research is focused on investigating the presence of the biases in LLM Overview systems and on biases exploitation to manipulate LLM Overview results. Here we train a small language model using reinforcement learning to rewrite search snippets to increase their likelihood of being preferred by an LLM Overview. Our experimental setup intentionally restricts the policy to operate only on snippets and limits reward-hacking strategies, reflecting realistic constraints of web search environments. The results prove that LLM Overview systems have biases and that reinforcement learning in most of the cases can optimize snippet's content to manipulate LLM Overview results. We also prove that LLM Overview selections are driven by comparative rather than absolute advantages among candidate sources. In addition, we examine safety aspects of LLM Overview manipulation possibilities and show that context poisoning attacks can lead to inaccurate or harmful results.

Figures

Figures reproduced from arXiv: 2605.00012 by Roman Smirnov.

**Figure 2.** Figure 2: RL training reward without length constrains. Only LLM Overview reward. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Figure 3: RL training reward with length reward component (0.2 weight) and LLM Overview reward. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

**Figure 4.** Figure 4: Trained policy evaluation results. average performance of the trained policy is shown in [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

**Figure 5.** Figure 5: GPT 4.1 mini - randomization robustness. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗

**Figure 6.** Figure 6: GPT 4.1 nano - randomization robustness. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗

**Figure 7.** Figure 7: GPT 5 mini - randomization robustness. 9 Conclusion In the research, we have shown that LLM have biases that can result in biased LLM Overview performance, our simplified LLM Overview exhibited strong biases. Following the proof of the biases we researched if these biases can help to manipulate LLM Overview results. We applied reinforcement learning to optimize the policy that can rewrite snippets to make… view at source ↗

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Optimizing Visibility in Generative Engines: A Critical Survey of Generative Engine Optimization (2023-2026)
cs.IR 2026-07 conditional novelty 5.0

A critical review of GEO research concludes that already-retrieved content can improve citation and use, but no tested technique reliably raises organic discoverability or downstream traffic across engines.

Reference graph

Works this paper leans on

13 extracted references · cited by 1 Pith paper

[1]

Google ai overviews publisher traffic decline analysis, 2026

ALMCORP. Google ai overviews publisher traffic decline analysis, 2026. Accessed: 2026-03-15

2026
[2]

Esci dataset, 2023

Amazon Science. Esci dataset, 2023. GitHub repository

2023
[3]

Generative engine optimization: How to dominate ai search, 2025

Mahe Chen, Xiaoxuan Wang, Kaiwen Chen, and Nick Koudas. Generative engine optimization: How to dominate ai search, 2025. 12

2025
[4]

Qlora: Efficient finetuning of quantized llms, 2023

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023

2023
[5]

google/gemma-3-1b-it, 2026

Google. google/gemma-3-1b-it, 2026

2026
[6]

Rewrite-to-rank: Optimizing ad visibility via retrieval-aware text rewriting, 2025

Chloe Ho, Ishneet Sukhvinder Singh, Diya Sharma, Tanvi Reddy Anumandla, Michael Lu, Vasu Sharma, and Kevin Zhu. Rewrite-to-rank: Optimizing ad visibility via retrieval-aware text rewriting, 2025

2025
[7]

intfloat/e5-small, 2026

intfloat. intfloat/e5-small, 2026. Accessed: 2026-03-20

2026
[8]

Understanding r1-zero-like training: A critical perspective, 2025

Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective, 2025

2025
[9]

Adversarial search engine optimization for large language models, 2024

Fredrik Nestaas, Edoardo Debenedetti, and Florian Tram` er. Adversarial search engine optimization for large language models, 2024

2024
[10]

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024

2024
[11]

What generative search engines like and how to optimize web content cooperatively, 2025

Yujiang Wu, Shanshan Zhong, Yubin Kim, and Chenyan Xiong. What generative search engines like and how to optimize web content cooperatively, 2025

2025
[12]

Justice or prejudice? quantifying biases in llm-as-a-judge, 2024

Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. Justice or prejudice? quantifying biases in llm-as-a-judge, 2024

2024
[13]

"" The user will provide a d i c t i o n a r y of search results in JSON format for search query \

Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, W...

2025

[1] [1]

Google ai overviews publisher traffic decline analysis, 2026

ALMCORP. Google ai overviews publisher traffic decline analysis, 2026. Accessed: 2026-03-15

2026

[2] [2]

Esci dataset, 2023

Amazon Science. Esci dataset, 2023. GitHub repository

2023

[3] [3]

Generative engine optimization: How to dominate ai search, 2025

Mahe Chen, Xiaoxuan Wang, Kaiwen Chen, and Nick Koudas. Generative engine optimization: How to dominate ai search, 2025. 12

2025

[4] [4]

Qlora: Efficient finetuning of quantized llms, 2023

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023

2023

[5] [5]

google/gemma-3-1b-it, 2026

Google. google/gemma-3-1b-it, 2026

2026

[6] [6]

Rewrite-to-rank: Optimizing ad visibility via retrieval-aware text rewriting, 2025

Chloe Ho, Ishneet Sukhvinder Singh, Diya Sharma, Tanvi Reddy Anumandla, Michael Lu, Vasu Sharma, and Kevin Zhu. Rewrite-to-rank: Optimizing ad visibility via retrieval-aware text rewriting, 2025

2025

[7] [7]

intfloat/e5-small, 2026

intfloat. intfloat/e5-small, 2026. Accessed: 2026-03-20

2026

[8] [8]

Understanding r1-zero-like training: A critical perspective, 2025

Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective, 2025

2025

[9] [9]

Adversarial search engine optimization for large language models, 2024

Fredrik Nestaas, Edoardo Debenedetti, and Florian Tram` er. Adversarial search engine optimization for large language models, 2024

2024

[10] [10]

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024

2024

[11] [11]

What generative search engines like and how to optimize web content cooperatively, 2025

Yujiang Wu, Shanshan Zhong, Yubin Kim, and Chenyan Xiong. What generative search engines like and how to optimize web content cooperatively, 2025

2025

[12] [12]

Justice or prejudice? quantifying biases in llm-as-a-judge, 2024

Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. Justice or prejudice? quantifying biases in llm-as-a-judge, 2024

2024

[13] [13]

"" The user will provide a d i c t i o n a r y of search results in JSON format for search query \

Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, W...

2025