Adversarial Attacks on LLM-as-a-Judge Systems: Insights from Prompt Injections

Dmitry Namiot; Narek Maloyan

Adversarial Attacks on LLM-as-a-Judge Systems: Insights from Prompt Injections

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2504.18333 v1 pith:PA263E2O submitted 2025-04-25 cs.CR cs.CL

Adversarial Attacks on LLM-as-a-Judge Systems: Insights from Prompt Injections

Narek Maloyan , Dmitry Namiot This is my paper

classification cs.CR cs.CL

keywords attackspromptpointcodefiftyfivegemmainjection

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

0 comments

read the original abstract

LLM as judge systems used to assess text quality code correctness and argument strength are vulnerable to prompt injection attacks. We introduce a framework that separates content author attacks from system prompt attacks and evaluate five models Gemma 3.27B Gemma 3.4B Llama 3.2 3B GPT 4 and Claude 3 Opus on four tasks with various defenses using fifty prompts per condition. Attacks achieved up to seventy three point eight percent success smaller models proved more vulnerable and transferability ranged from fifty point five to sixty two point six percent. Our results contrast with Universal Prompt Injection and AdvPrompter We recommend multi model committees and comparative scoring and release all code and datasets

discussion (0)

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

LLM-as-a-Judge for Evaluating System Responses in Conversational Music Recommendation
cs.IR 2026-07 conditional novelty 6.0

LLM judges agree moderately with human experts when scoring conversational music recommendation responses, outperform reference-based metrics, but are not reliable enough to replace human evaluation.
Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges
cs.CL 2026-05 conditional novelty 6.0

Per-bias selection of a cross-family LLM auditor lifts biased-judgment accuracy from 0.805/0.824 baselines to 0.884.
Large Language Models in Misinformation Ecosystems: Misuse, Defense, and Vulnerability
cs.CR 2026-07 conditional novelty 5.0

A role-layer survey unifies LLM misuse, LLM-based defense, and LLM-centric verification vulnerabilities across content, social, evidence, and workflow layers, then lists three open challenges.
Prompt Governance? On Governing Technologies Governed by Natural Language
cs.CY 2026-04 unverdicted novelty 4.0

Literature on system prompts for AI shows fragmented and contradictory claims that complicate policy efforts to use them as reliable governance mechanisms.