Prompt injection attacks can self-replicate across LLM agents in multi-agent systems, enabling data theft, misinformation, and system disruption while propagating silently.
hub Mixed citations
Deep reinforcement learning from human preferences
Mixed citation behavior. Most common role is background (60%).
49
Pith papers citing it
508
external citations · Pith
Background
60% of classified citations
abstract
For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than one percent of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach, we show that we can successfully train complex novel behaviors with about an hour of human time. These behaviors and environments are considerably more complex than any that have been previously learned from human feedback.
hub tools
citation-role summary
background 8
method 2
citation-polarity summary
representative citing papers
Promptbreeder evolves both task prompts and the mutation prompts that improve them using LLMs, outperforming Chain-of-Thought and Plan-and-Solve on arithmetic and commonsense reasoning benchmarks.
An unsupervised technique extracts latent yes-no knowledge from language model activations by locating a direction that satisfies logical consistency properties, outperforming zero-shot accuracy by 4% on average across models and datasets.
DecompRL is an RL method that learns modular code decomposition for LLMs, enabling exponential candidate generation via recombination to solve harder coding problems with lower GPU cost.
LLM multi-agent systems on lattices show bias-driven order-disorder crossovers instead of true phase transitions, with extracted effective couplings and fields serving as model-specific fingerprints.
Topology-enhanced alignment via persistent homology on trajectories outperforms standard SFT and DPO baselines on preference metrics for LLMs.
An identification theorem shows that a randomized experiment and simulator together recover causal model values from confounded logs, with logs used only afterward to reduce estimation error.
A new 7x4 taxonomy organizes agentic AI security threats by architectural layer and persistence timescale, revealing under-explored upper layers and missing defenses after surveying 116 papers.
HiPO improves LLM reasoning performance by optimizing preferences separately on response segments rather than entire outputs.
Alignment of vision-language models with human V1-V3 early visual cortex negatively predicts resistance to sycophantic gaslighting attacks.
Composable LoRA adapters can amplify or suppress OCEAN traits in LLMs, combine approximately additively, preserve moderate-scale capability, and move safety-relevant behaviours.
FPL trains a language-conditioned reward model from per-axis human preferences and a reward-conditioned policy, reporting 38-point average success gains over sparse-reward and binary-preference baselines on six manipulation tasks.
RAC is a closed-form bias correction for delayed rewards in RLHF that is unbiased under full mass reinjection of the delay kernel and reduces to V-trace with no delay.
Presents classical Õ(n²/ε^{1.5}) and quantum Õ(n/ε^{1.5}) query algorithms for ε-stationary points of twice-differentiable non-convex functions with Lipschitz gradient and Hessian via comparison oracles.
Fine-tuning VLMs with pairwise progress supervision from policy rollouts improves fine-grained failure detection and boosts robot manipulation success by 11% real-world and 5.9% in simulation.
A teacher-student reward model learns reasoning-conditioned score distributions for text-to-image images, yielding ~89% preference accuracy and a 41% net human-preference gain when used for generator optimization.
Prefix filters learned by the Palla algorithm capture LLM error patterns and enable constrained sampling that boosts TypeScript compile rates by over 60% for Qwen2.5-1.5B to match larger models.
Agent-directed tree search improves LLM performance on Lean formal verification tasks, with context-based orchestration solving more intermediate specs at lower token cost than baseline agents.
A latent mediation framework with sparse autoencoders enables non-additive token-level influence attribution in LLMs by learning orthogonal features and back-propagating attributions.
Mechanistic analysis of six preference optimization methods reveals distinct geometric shifts in model representations, with KTO/GRPO enhancing separability while DPO/ORPO degrade it.
The power distribution is the target of power sampling, the closed-form solution to self-reward KL-regularized RL, and the basis for power self-distillation that matches sampling performance at lower cost.
Relax is a new RL training engine with omni-native design and async execution that delivers up to 2x speedups over baselines like veRL while converging to equivalent reward levels on Qwen3 models.
A Dirichlet-prior Bayesian estimator for model success probability replaces Pass@k, delivering faster-converging and more stable rankings with credible intervals on math benchmarks.
VLA-RL applies online RL to pretrained VLAs, yielding a 4.5% gain over strong baselines on 40 LIBERO manipulation tasks and matching commercial models like π₀-FAST.
citing papers explorer
-
Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems
Prompt injection attacks can self-replicate across LLM agents in multi-agent systems, enabling data theft, misinformation, and system disruption while propagating silently.
-
Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution
Promptbreeder evolves both task prompts and the mutation prompts that improve them using LLMs, outperforming Chain-of-Thought and Plan-and-Solve on arithmetic and commonsense reasoning benchmarks.
-
Discovering Latent Knowledge in Language Models Without Supervision
An unsupervised technique extracts latent yes-no knowledge from language model activations by locating a direction that satisfies logical consistency properties, outperforming zero-shot accuracy by 4% on average across models and datasets.
-
DecompRL: Solving Harder Problems by Learning Modular Code Generation
DecompRL is an RL method that learns modular code decomposition for LLMs, enabling exponential candidate generation via recombination to solve harder coding problems with lower GPU cost.
-
Collective Alignment in LLM Multi-Agent Systems: Disentangling Bias from Cooperation via Statistical Physics
LLM multi-agent systems on lattices show bias-driven order-disorder crossovers instead of true phase transitions, with extracted effective couplings and fields serving as model-specific fingerprints.
-
Topology-Enhanced Alignment for Large Language Models: Trajectory Topology Loss and Topological Preference Optimization
Topology-enhanced alignment via persistent homology on trajectories outperforms standard SFT and DPO baselines on preference metrics for LLMs.
-
The Partial Testimony of Logs: Evaluation of Language Model Generation under Confounded Model Choice
An identification theorem shows that a randomized experiment and simulator together recover causal model values from confounded logs, with logs used only afterward to reduce estimation error.
-
A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Framework
A new 7x4 taxonomy organizes agentic AI security threats by architectural layer and persistence timescale, revealing under-explored upper layers and missing defenses after surveying 116 papers.
-
HiPO: Hierarchical Preference Optimization for Adaptive Reasoning in LLMs
HiPO improves LLM reasoning performance by optimizing preferences separately on response segments rather than entire outputs.
-
Gaslight, Gatekeep, V1-V3: Early Visual Cortex Alignment Shields Vision-Language Models from Sycophantic Manipulation
Alignment of vision-language models with human V1-V3 early visual cortex negatively predicts resistance to sycophantic gaslighting attacks.
-
Persona Cartography: Charting Language Model Personality Traits in Weight Space
Composable LoRA adapters can amplify or suppress OCEAN traits in LLMs, combine approximately additively, preserve moderate-scale capability, and move safety-relevant behaviours.
-
Freeform Preference Learning for Robotic Manipulation
FPL trains a language-conditioned reward model from per-axis human preferences and a reward-conditioned policy, reporting 38-point average success gains over sparse-reward and binary-preference baselines on six manipulation tasks.
-
Retroactive Advantage Correction: Closed-Form V-Trace Bias Correction for Delay-Aware RLHF
RAC is a closed-form bias correction for delayed rewards in RLHF that is unbiased under full mass reinjection of the delay kernel and reduces to V-trace with no delay.
-
Finding Stationary Points by Comparisons
Presents classical Õ(n²/ε^{1.5}) and quantum Õ(n/ε^{1.5}) query algorithms for ε-stationary points of twice-differentiable non-convex functions with Lipschitz gradient and Hessian via comparison oracles.
-
Robot Critics that Sweat the Small Stuff
Fine-tuning VLMs with pairwise progress supervision from policy rollouts improves fine-grained failure detection and boosts robot manipulation success by 11% real-world and 5.9% in simulation.
-
Z-Reward: Beyond Scalar Rewards by Internalizing Reasoning into Score Distributions
A teacher-student reward model learns reasoning-conditioned score distributions for text-to-image images, yielding ~89% preference accuracy and a 41% net human-preference gain when used for generator optimization.
-
Learning the Error Patterns of Language Models
Prefix filters learned by the Palla algorithm capture LLM error patterns and enable constrained sampling that boosts TypeScript compile rates by over 60% for Qwen2.5-1.5B to match larger models.
-
Automating Formal Verification with Agent-Guided Tree Search
Agent-directed tree search improves LLM performance on Lean formal verification tasks, with context-based orchestration solving more intermediate specs at lower token cost than baseline agents.
-
Correcting Influence: Unboxing LLM Outputs with Orthogonal Latent Spaces
A latent mediation framework with sparse autoencoders enables non-additive token-level influence attribution in LLMs by learning orthogonal features and back-propagating attributions.
-
Mechanistic Analysis of Alignment Algorithms in Language Models
Mechanistic analysis of six preference optimization methods reveals distinct geometric shifts in model representations, with KTO/GRPO enhancing separability while DPO/ORPO degrade it.
-
Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation
The power distribution is the target of power sampling, the closed-form solution to self-reward KL-regularized RL, and the basis for power self-distillation that matches sampling performance at lower cost.
-
Relax: An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scale
Relax is a new RL training engine with omni-native design and async execution that delivers up to 2x speedups over baselines like veRL while converging to equivalent reward levels on Qwen3 models.
-
Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation
A Dirichlet-prior Bayesian estimator for model success probability replaces Pass@k, delivering faster-converging and more stable rankings with credible intervals on math benchmarks.
-
VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
VLA-RL applies online RL to pretrained VLAs, yielding a 4.5% gain over strong baselines on 40 LIBERO manipulation tasks and matching commercial models like π₀-FAST.
-
Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
Repeated sampling scales problem coverage log-linearly with sample count, improving SWE-bench Lite performance from 15.9% to 56% using 250 samples.
-
Improve Mathematical Reasoning in Language Models by Automated Process Supervision
OmegaPRM automates collection of 1.5 million process supervision labels via binary-search MCTS, raising Gemini Pro math accuracy from 51% to 69.4% on MATH500 and Gemma2 27B from 42.3% to 58.2%.
-
Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models
Sparse feature circuits are introduced as interpretable causal subnetworks in language models, supporting unsupervised discovery of thousands of circuits and a method called SHIFT to improve classifier generalization by ablating irrelevant features.
-
Simple synthetic data reduces sycophancy in large language models
Scaling and instruction tuning increase sycophancy in LLMs on opinion and fact tasks, but a synthetic data fine-tuning intervention reduces it on held-out prompts.
-
A General Language Assistant as a Laboratory for Alignment
Ranked preference modeling outperforms imitation learning for language model alignment and scales more favorably with model size.
-
Alignment Plausibility: A New Standard for Assuring AI in Healthcare
Alignment plausibility—evidence that an AI system's values, training, and oversight cohere with safe positive health outcomes—should be the regulatory analogue of biological plausibility for LLMs in healthcare.
-
Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF
Two plug-and-play strategies — per-timestep advantage weighting and advantage-based trajectory replay — improve diffusion RLHF sample efficiency up to 6× across five reward functions.
-
Pessimism's Paradox: Conservative Offline Training Amplifies Reward Hacking During Online Adaptation in Reasoning Models
Higher conservatism in offline DPO training of Qwen3-14B monotonically increases reward-hacking damage (Goodhart gap AUGC) during online adaptation on GSM8K.
-
Designing Reward Signals for Portable Query Generation: A Case Study in Industrial Semantic Job Search
Empirical study of RLAIF for portable query generation finds reward shaping controls performance more than optimizer choice and a rule-based reward floor yields +0.147 quality gain.
-
Speculative Decoding at Temperature Zero: A Scoped Safety-Invariance Screen with a 48,072-Sample Expansion
No detectable safety divergence between target-only and speculative decoding at temperature zero under TAIS criteria on 48,072 samples across safety benchmarks.
-
Explanation Quality Assessment as Ranking with Listwise Rewards
Explanation quality assessment is recast as ranking with listwise and pairwise losses that outperform regression, allow small models to match large ones on curated data, and enable stable convergence in reinforcement learning.
-
From Perception to Autonomous Computational Modeling: A Multi-Agent Approach
A multi-agent LLM framework autonomously completes the full computational mechanics pipeline from a photograph to a code-compliant engineering report on a steel L-bracket example.
-
Failure Modes of Maximum Entropy RLHF
Derives SimPO from MaxEnt RL and reports that MaxEnt RL in online RLHF exhibits frequent overoptimization and unstable KL dynamics across scales, unlike stable KL-constrained baselines.
-
Probabilistic Modeling of Latent Agentic Substructures in Deep Neural Networks
Proposes a probabilistic framework for latent agentic substructures in DNNs using log-score utilities and log pooling, with proofs on unanimity and an application to persona emergence in LLM alignment.
-
Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning
Proposes token-significance and dynamic length rewards in RL to reduce LLM response length while preserving or improving reasoning correctness across benchmarks.
-
Machine-Coached Policy Revision in Adaptive Agent-Based Regulatory Simulation: A Controller-Level Contestability Layer
Introduces a defeasible rule-based coaching layer that converts diagnostic failures into policy rule revisions in adaptive agent-based regulatory simulations, demonstrated on an emissions-regulation ABM.
-
Uncertainty-Aware LLM-Guided Policy Shaping for Sparse-Reward Reinforcement Learning
ULPS integrates A*-generated symbolic trajectories, fine-tuned BERT priors, MC dropout uncertainty, and entropy-based blending into PPO, reporting over 9% accuracy gains and better sample efficiency on the MiniGridUnlockPickup benchmark.
-
Agentic Safety is an Epistemic Property, Not a Behavioral One
The paper reframes agentic safety as an epistemic property defined by teachability—the capacity to preserve future corrective leverage—rather than a behavioral property of the current policy.
-
Pluralistic-Alignment Urbanism: Operationalizing a Right to AI for Inclusive Public Space
Introduces PAU as a governance architecture for municipal AI in public spaces, informed by case studies on subgroup-aware scaling (R2=0.89) and pluralistic preference data that treats neutrality as indeterminacy.
-
EyeMulator: Improving Code Language Models by Mimicking Human Visual Attention
The abstract and the full text are two different papers: EyeMulator (code LMs with eye-tracking) versus Walk-on-Interfaces (a PDE Monte Carlo method), so the submission is internally inconsistent.
-
Civilizational Metamaterials: Engineering Coordination Under Capability Gradients and Structural Turbulence
Introduces phenomenological model R_eff = β(1-ρ)(1-τ)(1-γρτ) for coordination under AGI decision velocity, with phase transition and proposed randomized trial.
-
An Overview of Catastrophic AI Risks
The paper categorizes sources of catastrophic AI risks into malicious use, AI race, organizational risks, and rogue AIs, providing illustrative stories and mitigation suggestions for each.
-
The Hitchhiker's Guide to Agentic AI: From Foundations to Systems
A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.
- Efficient Safety Alignment of Language Models via Latent Personality Traits
- Decodable but Not Faithful: Coupling Natural-Language Rationales to Programmatic Verifiers