pith. sign in

arxiv: 2605.29678 · v1 · pith:JDHPMCZPnew · submitted 2026-05-28 · 💻 cs.CL

Spurious Prompts: Can Irrelevant Prompts Steer Large Language Models?

Pawel Batorski , Abtin Pourhadi , Jerzy Sarosiek , Przemyslaw Spurek , Paul Swoboda This is my paper

Pith reviewed 2026-06-29 07:47 UTC · model grok-4.3

classification 💻 cs.CL
keywords spurious promptsprompt sensitivitylarge language modelsblack-box searchunintended behaviorsreasoning benchmarksquestion answering
0
0 comments X

The pith

Prompts with no semantic link to the task can still steer large language models toward higher scores or specific errors.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that prompts unrelated in meaning to a given task can nevertheless raise accuracy on reasoning and question-answering benchmarks or push models into repeatable unwanted patterns. A sympathetic reader would care because this shows model behavior depends on more than the intended content of instructions. The authors introduce a black-box search method that finds such prompts automatically and demonstrate the effect across model sizes from 0.8B to 27B parameters and multiple families. They also show the same prompts can force outputs such as always choosing the first option or returning even, prime, or small numbers.

Core claim

Spurious prompts, defined as prompts that are semantically unrelated to the task, can improve performance on reasoning and question-answering benchmarks, often matching or outperforming standard prompting baselines and task-aware prompt optimization. The same prompts can also steer models toward unintended behaviors such as repeatedly selecting the first answer option, producing incorrect answers, or returning an even, prime or small number without any explicit instruction to do so.

What carries the argument

A black-box search procedure that discovers spurious prompts capable of steering model outputs despite having no semantic connection to the task.

If this is right

  • Spurious prompts can raise benchmark scores without any task-relevant content being supplied.
  • Models can be induced to select the first listed answer option on every question.
  • Models can be made to output only even numbers, only prime numbers, or only small numbers in their responses.
  • The steering effect appears across models from 0.8B to 27B parameters in three different families.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Safety filters that scan only for task-related language may miss steering signals carried by unrelated text.
  • Prompt engineering workflows could be simplified or replaced by broad random searches if the effect proves general.
  • Additional context added to prompts for other reasons might unintentionally alter model behavior on the main task.

Load-bearing premise

The prompts located by the search are genuinely unrelated to the task rather than carrying hidden task information through the search process itself.

What would settle it

Showing that every high-performing spurious prompt identified by the procedure contains subtle task-relevant cues that disappear when those cues are removed or rephrased.

Figures

Figures reproduced from arXiv: 2605.29678 by Abtin Pourhadi, Jerzy Sarosiek, Paul Swoboda, Pawel Batorski, Przemyslaw Spurek.

Figure 1
Figure 1. Figure 1: Right: High-level illustration of Spurious Prompting. We search for prompts that are unrelated to the target task but can nevertheless solve it. Left: Average accuracy over seven benchmarks comparing Spurious Prompting with PromptWizard and zero-shot chain-of-thought prompting across four models ranging from 0.8B to 27B parameters. Spurious Prompting achieves performance comparable to, and in some cases be… view at source ↗
Figure 2
Figure 2. Figure 2: Top: Overview of our fully black-box search procedure. An LLM generator first proposes candidate prompts and is explicitly instructed to make them unrelated to the target task. These candidates are then passed to a prompt validator, which filters out prompts that contain task-relevant content. The remaining prompts are evaluated on a subset of the training data, after which the top-K prompts are mutated an… view at source ↗
Figure 4
Figure 4. Figure 4: Mean cosine similarity between prompt text [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Average prompt length in tokens for spuri [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Cumulative ablation of spurious-prompt com [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
read the original abstract

Large language models are highly sensitive to prompts, but this sensitivity is usually studied through task-relevant instructions, demonstrations, or reasoning cues. In this paper, we study a different form of prompt sensitivity: whether prompts that are semantically unrelated to the task can nevertheless steer model behavior. We call them spurious prompts and show their surprising efficacy. We also propose a simple black-box search procedure for discovering them. Across reasoning and question-answering benchmarks, using models ranging from 0.8B to 27B parameters and spanning three model families, we show that spurious prompts can improve performance, often matching or outperforming standard prompting baselines and task-aware prompt optimization. We further show that they can steer models toward unintended behaviors, such as repeatedly selecting the first answer option, producing incorrect answers, returning an even, prime or small number without explicitly instructing the model to do so. These findings reveal a new kind of prompt sensitivity: LLMs can be systematically steered by prompts that are unrelated to the task they are asked to solve. Our code is available at https://github.com/Batorskq/spurious

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The paper claims that semantically unrelated 'spurious' prompts, discovered via a black-box search procedure, can steer LLMs on reasoning and QA tasks to improve performance (often matching or exceeding standard prompting and task-aware optimization) and induce unintended behaviors such as first-option bias or generating even/prime/small numbers, across models from 0.8B to 27B parameters in three families. The work positions this as evidence of a new form of prompt sensitivity.

Significance. If the results hold under rigorous verification of prompt unrelatedness, the findings would indicate a previously understudied vulnerability in LLM prompt sensitivity with implications for robustness and safety. Credit is due for releasing reproducible code and conducting experiments across multiple model scales and families.

major comments (3)
  1. [Abstract and Methods] Abstract and search procedure description: no objective, reproducible metric (e.g., embedding cosine threshold, lexical overlap filter, or blinded ratings) is provided to confirm that discovered prompts are verifiably semantically unrelated to the task rather than carrying latent task information via the search dynamics. This assumption is load-bearing for interpreting the results as evidence of 'spurious' steering.
  2. [Results] Results section: performance claims lack error bars, standard deviations, or statistical significance tests, and dataset details (exact benchmarks, example counts, train/test splits) are insufficient to evaluate whether gains are reliable or to rule out leakage.
  3. [Experimental Setup] Experimental setup: the black-box search procedure is not described with enough detail to verify it avoids task leakage or post-hoc selection bias, undermining the central claim that unrelated prompts drive the observed effects.
minor comments (1)
  1. [Abstract] The abstract states experiments span 'three model families' without naming them; this should be specified for clarity.

Simulated Author's Rebuttal

3 responses · 0 unresolved

Thank you for the constructive comments. We address each of the major comments point-by-point below, indicating the revisions we will make to the manuscript.

read point-by-point responses

  1. Referee: [Abstract and Methods] Abstract and search procedure description: no objective, reproducible metric (e.g., embedding cosine threshold, lexical overlap filter, or blinded ratings) is provided to confirm that discovered prompts are verifiably semantically unrelated to the task rather than carrying latent task information via the search dynamics. This assumption is load-bearing for interpreting the results as evidence of 'spurious' steering.

    Authors: We agree that an objective metric for confirming semantic unrelatedness is important for the central claim. In the revised version, we will add a dedicated subsection in the Methods describing our verification process. This will include computing embedding cosine similarities using a pre-trained sentence transformer (e.g., all-MiniLM-L6-v2) between the spurious prompts and task descriptions, reporting that similarities are below 0.15 on average. We will also provide results from a small-scale blinded human evaluation where raters assess relatedness on a 1-5 scale, with average scores below 2. This will be applied to the discovered prompts across tasks. revision: yes

  2. Referee: [Results] Results section: performance claims lack error bars, standard deviations, or statistical significance tests, and dataset details (exact benchmarks, example counts, train/test splits) are insufficient to evaluate whether gains are reliable or to rule out leakage.

    Authors: We acknowledge the need for greater statistical transparency and dataset clarity. The revised manuscript will include error bars representing standard deviation over 3-5 independent runs with different random seeds for all reported accuracies. We will add statistical significance testing using paired t-tests or Wilcoxon tests between conditions. Additionally, we will expand the Experimental Setup section with a table detailing each benchmark (e.g., GSM8K with 1319 test examples, using the standard test split with no training data leakage as we do not fine-tune). revision: yes

  3. Referee: [Experimental Setup] Experimental setup: the black-box search procedure is not described with enough detail to verify it avoids task leakage or post-hoc selection bias, undermining the central claim that unrelated prompts drive the observed effects.

    Authors: We appreciate this feedback on the search procedure description. In the revision, we will provide a more detailed algorithmic description of the black-box search, including pseudocode, the exact optimization method (e.g., evolutionary search with population size, mutation rate), the fitness function (task accuracy on a small validation set), and explicit steps to avoid leakage such as generating prompts from a general vocabulary without task-specific terms and using a separate validation set for search that is disjoint from the test set. We will also discuss and mitigate post-hoc selection bias by reporting performance on fully held-out test data not used in any selection. revision: yes

Circularity Check

0 steps flagged

No significant circularity in empirical black-box study

full rationale

The paper reports direct experimental outcomes from a black-box search procedure and evaluations on reasoning/QA benchmarks across model sizes and families. No derivations, equations, fitted parameters, or first-principles claims are present that could reduce results to inputs by construction. The central findings (performance gains and steering effects from spurious prompts) are presented as measured empirical results rather than derived quantities. The paper is self-contained against external benchmarks with code released for reproduction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that LLMs respond to prompt wording independently of semantic content and on the empirical claim that the black-box search finds truly spurious prompts.

axioms (1)
  • domain assumption LLMs remain sensitive to prompt text even when that text carries no semantic relation to the task
    This is the core premise being tested and is invoked throughout the abstract to frame the results.

pith-pipeline@v0.9.1-grok · 5738 in / 1181 out tokens · 26744 ms · 2026-06-29T07:47:18.742031+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

3 extracted references

  1. [1]

    InThe Twelfth Inter- national Conference on Learning Representations

    Let’s verify step by step. InThe Twelfth Inter- national Conference on Learning Representations. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024. AutoDAN: Generating stealthy jailbreak prompts on aligned large language models. InThe Twelfth International Conference on Learning Repre- sentations. Sheng Lu, Hendrik Schuff, and Iryna Gurevych. 2024....

    2024

  2. [2]

    Albert Webson and Ellie Pavlick

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.Ad- vances in Neural Information Processing Systems, 37:95266–95290. Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning of their prompts? InProceedings of the 2022 Conference of the North American Chapter of the Association for Com...

    2022

  3. [3]

    this should not help with GSM8K,

    Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. InInternational Conference on Learning Representations, volume 2024, pages 53902–53922. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In International conference on machine learning, pages 1...

    2024