MaMa: A Game-Theoretic Approach for Designing Safe Agentic Systems

Adish Singla; Goran Radanovic; Jonathan N\"other

arxiv: 2602.04431 · v2 · pith:ED66SJSDnew · submitted 2026-02-04 · 💻 cs.LG · cs.GT

MaMa: A Game-Theoretic Approach for Designing Safe Agentic Systems

Jonathan N\"other , Adish Singla , Goran Radanovic This is my paper

Pith reviewed 2026-05-25 07:23 UTC · model grok-4.3

classification 💻 cs.LG cs.GT

keywords multi-agent systemsLLM safetyadversarial robustnessgame theoryStackelberg gamesagentic systemsautomated design

0 comments

The pith

MaMa designs LLM agent systems via iterative Meta-Agent and Meta-Adversary feedback to resist worst-case compromises while preserving task performance.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that modeling the automated design of safe multi-agent LLM systems as a Stackelberg game between a Meta-Agent designer and a Meta-Adversary allows systems to be built that remain safe even when some agents are compromised. The Meta-Agent proposes designs while the Meta-Adversary uses LLM-based search to identify the strongest attacks by choosing which agents to compromise, and the process repeats to improve robustness. A sympathetic reader would care because LLM-based agent teams carry safety risks from individual failures or adversarial behavior, and this method automates finding designs that defend against those risks without manual intervention. Empirical results indicate the resulting systems hold up under worst-case attacks, match the performance of purely task-focused designs, and extend to new adversaries or different LLMs.

Core claim

The central claim is that MaMa, an algorithm inspired by Stackelberg security games, uses LLM-based adversarial search in which the Meta-Agent iteratively proposes agentic system designs and receives feedback based on the strongest attacks discovered by the Meta-Adversary, producing systems that defend against worst-case attacks while maintaining performance comparable to systems optimized solely for task success, and that these systems generalize to stronger adversaries as well as ones with different attack objectives or underlying LLMs.

What carries the argument

MaMa is the iterative algorithm that alternates between the Meta-Agent proposing agentic system designs and the Meta-Adversary selecting and compromising a subset of agents to minimize safety, using LLM-based adversarial search to identify the attacks.

If this is right

Systems designed with MaMa defend against worst-case attacks.
Task performance remains comparable to systems optimized only for success without safety considerations.
The systems generalize to stronger adversaries than those used in design.
They also generalize to adversaries with different attack objectives or different underlying LLMs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same iterative designer-adversary loop could be tested on non-LLM multi-agent systems to check if the robustness benefit transfers.
It raises the possibility that game-theoretic automation could reduce reliance on post-hoc safety patches in deployed agent teams.

Load-bearing premise

The LLM-based adversarial search performed by the Meta-Adversary is able to discover attacks that are representative of the true worst-case compromises an external attacker could mount.

What would settle it

Showing that a MaMa-designed system is successfully compromised by a stronger search procedure or by human red-teaming that finds attacks the Meta-Adversary missed during design would falsify the robustness claim.

read the original abstract

LLM-based multi-agent systems have demonstrated impressive capabilities, but they also introduce significant safety risks when individual agents fail or behave adversarially. In this work, we study the automated design of agentic systems that remain safe even when a subset of agents is compromised. Inspired by Stackelberg security games, we formalize this problem as a game between a system designer (the Meta-Agent) and a best-responding Meta-Adversary that selects and compromises a subset of agents to minimize safety. We propose Meta-Adversary-Meta-Agent (MaMa), a novel algorithm inspired by this formalization for automatically designing safe agentic systems. Our approach uses LLM-based adversarial search, where the Meta-Agent iteratively proposes system designs and receives feedback based on the strongest attacks discovered by the Meta-Adversary. Empirical evaluations across diverse environments show that systems designed with MaMa consistently defend against worst-case attacks while maintaining performance comparable to systems optimized solely for task success. Moreover, the resulting systems generalize to stronger adversaries, as well as ones with different attack objectives or underlying LLMs, demonstrating robust safety beyond the training setting.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

MaMa frames safe multi-agent LLM design as a Stackelberg game with LLM-driven adversarial search, but the safety results rest on whether that search actually finds hard attacks.

read the letter

The main takeaway is that this paper turns the problem of building robust agentic systems into an iterative game where one LLM proposes designs and another tries to break them by compromising agents. The MaMa algorithm uses that loop to produce systems that hold up in their reported tests while keeping task performance intact, and it claims the designs transfer to stronger or different adversaries and models. That combination of Stackelberg security games with LLM-based search is the concrete new piece relative to earlier agent-safety work. The formalization is clean and the motivation from real deployment risks is straightforward. The experiments are described at a high level as covering diverse environments with generalization checks, which is better than pure theory. The soft spot is exactly the one the stress-test flags: the meta-adversary is itself an LLM heuristic running inside the same model family, so it can miss attacks that require precise coordination, non-text exploits, or details not in the prompt. If those missed attacks exist, the “defends against worst-case” claim and the generalization results lose force. The abstract does not show how they validated that the discovered attacks were near-optimal or compared them to human or external red-teaming. Without that, the empirical support stays provisional. This is aimed at researchers building or evaluating multi-agent LLM deployments who need automated design tools. It is coherent on its own terms and engages the literature honestly, so it deserves a serious referee even if revisions will be needed on the attack-quality checks.

Referee Report

2 major / 2 minor

Summary. The paper formalizes the design of safe LLM-based multi-agent systems as a Stackelberg security game between a Meta-Agent (system designer) and a Meta-Adversary that compromises a subset of agents. It introduces the MaMa algorithm, which uses iterative LLM-based adversarial search: the Meta-Agent proposes designs and refines them based on the strongest attacks found by the Meta-Adversary. The central empirical claim is that MaMa-designed systems defend against worst-case attacks while preserving task performance and generalize to stronger adversaries, different attack objectives, and different underlying LLMs.

Significance. If the empirical results hold under rigorous validation of the adversary, the work offers a principled, automated method for safety in agentic systems that goes beyond ad-hoc prompting. The Stackelberg framing and iterative feedback loop are clear strengths; the generalization experiments, if they include controls for adversary strength, would be a notable contribution to robust multi-agent LLM design.

major comments (2)

[Abstract] Abstract: the headline claim that MaMa systems 'defend against worst-case attacks' is load-bearing for all empirical conclusions, yet it rests on the unverified assumption that the LLM-driven Meta-Adversary search reliably approximates true worst-case compromises. The description provides no comparison to human-crafted attacks, formal attack models, or exhaustive search baselines that would substantiate this.
[Abstract] Abstract (generalization paragraph): the claim that systems 'generalize to stronger adversaries' cannot be evaluated without evidence that the training-time attacks were near-optimal; if the Meta-Adversary is itself a heuristic within the same model family, stronger external attacks could lie outside the discovered distribution, rendering the generalization result circular.

minor comments (2)

The abstract refers to 'diverse environments' and 'comparable performance' without naming the environments, metrics, or baseline systems used; these details are needed for reproducibility even at the high level.
Notation for the game (Meta-Agent vs. Meta-Adversary) is introduced without an explicit payoff matrix or equilibrium definition in the provided text; a short formal section would clarify the Stackelberg structure.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. We address the two major comments on the abstract claims point by point, proposing targeted revisions to improve precision without misrepresenting the work.

read point-by-point responses

Referee: [Abstract] Abstract: the headline claim that MaMa systems 'defend against worst-case attacks' is load-bearing for all empirical conclusions, yet it rests on the unverified assumption that the LLM-driven Meta-Adversary search reliably approximates true worst-case compromises. The description provides no comparison to human-crafted attacks, formal attack models, or exhaustive search baselines that would substantiate this.

Authors: We agree that 'worst-case' in the abstract is scoped to attacks generated by the Meta-Adversary within the Stackelberg formulation rather than an absolute worst-case over all possible compromises. The manuscript does not contain human-crafted attack comparisons or exhaustive baselines, which is a limitation of the current LLM-centric evaluation. We will revise the abstract to explicitly qualify the claim as defending against the strongest attacks discovered by the Meta-Adversary and add a brief limitations paragraph discussing the approximation and suggesting future validation work. This is a clarification of scope. revision: yes
Referee: [Abstract] Abstract (generalization paragraph): the claim that systems 'generalize to stronger adversaries' cannot be evaluated without evidence that the training-time attacks were near-optimal; if the Meta-Adversary is itself a heuristic within the same model family, stronger external attacks could lie outside the discovered distribution, rendering the generalization result circular.

Authors: We acknowledge the risk of circularity if the Meta-Adversary is not near-optimal. The iterative search is intended to produce progressively stronger attacks, and the paper reports that defense improves accordingly. To address the concern, we will revise the generalization paragraph to specify that the systems generalize to stronger adversaries generated by the same LLM-based method and model family. We will also add analysis showing attack strength increases across iterations. This tempers the claim to match the experimental setting. revision: partial

Circularity Check

0 steps flagged

No circularity; empirical claims rest on external evaluations, not self-referential derivation.

full rationale

The paper formalizes a Stackelberg-inspired game between Meta-Agent and Meta-Adversary, then proposes the MaMa algorithm that iterates LLM-based proposals and attacks. The central claims are supported by empirical evaluations across environments showing defense against discovered attacks and generalization. No equations, fitted parameters, or self-citations are presented that reduce the result to its own inputs by construction. The derivation chain is algorithmic and externally validated via experiments rather than tautological.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract provides no explicit free parameters, axioms, or invented entities; ledger is therefore empty pending full text.

pith-pipeline@v0.9.0 · 5731 in / 1016 out tokens · 24319 ms · 2026-05-25T07:23:10.689782+00:00 · methodology

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Stable Agentic Control: Tool-Mediated LLM Architecture for Autonomous Cyber Defense
cs.AI 2026-05 unverdicted novelty 6.0 partial

Tool-mediated LLM agents with deterministic tools and a machine-checked Lyapunov certificate achieve stable control in cyber defense, reducing attacker game value by 59% on real attack graphs.