Pith. sign in

REVIEW 5 cited by

Automatically Auditing Large Language Models via Discrete Optimization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2303.04381 v1 pith:Z7WJ3OD2 submitted 2023-03-08 cs.LG cs.CL

Erik Jones , Anca Dragan , Aditi Raghunathan , Jacob Steinhardt This is my paper · ORCID
classification cs.LGcs.CL
keywords modelsoptimizationauditingautomaticallydiscreteinputslanguagebarack
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Auditing large language models for unexpected behaviors is critical to preempt catastrophic deployments, yet remains challenging. In this work, we cast auditing as an optimization problem, where we automatically search for input-output pairs that match a desired target behavior. For example, we might aim to find a non-toxic input that starts with "Barack Obama" that a model maps to a toxic output. This optimization problem is difficult to solve as the set of feasible points is sparse, the space is discrete, and the language models we audit are non-linear and high-dimensional. To combat these challenges, we introduce a discrete optimization algorithm, ARCA, that jointly and efficiently optimizes over inputs and outputs. Our approach automatically uncovers derogatory completions about celebrities (e.g. "Barack Obama is a legalized unborn" -> "child murderer"), produces French inputs that complete to English outputs, and finds inputs that generate a specific name. Our work offers a promising new tool to uncover models' failure-modes before deployment.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. JailbreaksOverTime: Detecting Jailbreak Attacks Under Distribution Shift

    cs.CR 2025-04 conditional novelty 7.0 of 10

    Jailbreak detection models drift over time, and a weekly self-trained detector plus an unsupervised behavioral monitor can keep false negatives near 0.3 to 0.4 percent at a 0.1 to 1 percent false positive rate.

  2. Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning

    cs.LG 2024-12 conditional novelty 6.0 of 10

    An automated red-teaming method that uses LLM-generated per-goal rewards and multi-step RL with a style-diversity reward to produce diverse and effective attacks on language models.

  3. InfoFlood: Jailbreaking Large Language Models with Information Overload

    cs.CR 2025-06 conditional novelty 5.0 of 10

    InfoFlood claims near-perfect jailbreak success on four frontier LLMs by rewriting harmful queries into verbose academic prose with fake citations, past-tense framing, and ethical disclaimers, without adversarial suffixes.

  4. Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignment

    cs.CL 2025-01 conditional novelty 5.0 of 10

    Fine-tuning and preference alignment of LLM-as-a-judge guardrails on 400 examples yields large F1 improvements over base models and beats LlamaGuard-2 on the authors' test set.

  5. PRM-Free Security Alignment of Large Models via Red Teaming and Adversarial Training

    cs.CR 2025-07 reject novelty 3.0 of 10

    A PRM-free alignment pipeline combining genetic algorithm red teaming and multi-objective adversarial training is claimed to beat PRM-based methods at 61% lower cost, but the experiments are unverifiable.

Pith tools