{"id":"42c4e399-fd9e-4121-95b9-d3dea504df4b","arxiv_id":"2605.30854","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"SEPO adds penalties for exploitability, collusion, and externalities to GRPO training of LM agents, yielding zero exploit advantage in Kuhn Poker and safety gains in four of five strategic domains.","lead":"The paper proposes Safe Equilibrium Policy Optimization (SEPO), which augments standard RL rewards for language model agents with explicit penalties for exploitability, collusion risk, and externality costs. A smart generalist might read it to see one concrete way to reduce harmful strategic behaviors when AI agents interact in games or negotiations.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"The reader's identified assumption is the exact point the manuscript tests and claims to resolve through ablation; with that addressed and artifacts released, it does not remain load-bearing for the central claim.","tokens_in":1785,"tokens_out":232,"duration_ms":16573,"concrete_test":"Re-run the GRPO ablation from the paper (shared constant vs. per-rollout exploit penalty) on the released code for one domain; confirm that only the per-rollout variant produces non-zero policy gradients and that the reported Kuhn Poker exploit-pool advantage remains zero under the same evaluation protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper directly addresses the gradient-vanishing concern via ablation experiments demonstrating that a shared constant penalty cancels under GRPO normalization while per-rollout exploitability computation does not. The central empirical claims (zero exploit-pool advantage in Kuhn Poker, safety improvements across domains, correction of SFT over-cooperation) rest on this mechanism plus released code and datasets; no internal inconsistency or unaddressed measurement gap is evident from the provided description.","agreement_with_reader":"disagree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes Safe Equilibrium Policy Optimization (SEPO), an objective that augments expected payoff with explicit penalties on exploitability, collusion risk, and externality cost for language-model agents in multi-agent settings. SEPO is implemented as a reward for Group Relative Policy Optimization (GRPO) on Gemma-4 E4B-it and Qwen-3.5-4B after SFT, and evaluated on five domains (Iterated Prisoner's Dilemma, repeated auctions, two negotiation variants, Kuhn Poker). The central claims are that SEPO achieves zero exploit-pool advantage in Kuhn Poker for both models, improves safety metrics over the base model in four domains, corrects SFT-induced over-cooperation, and that per-rollout exploitability computation is required because a shared constant penalty cancels under GRPO advantage normalization. Code and SFT datasets are released.","tokens_in":1872,"tokens_out":542,"duration_ms":18163,"significance":"If the results hold, SEPO supplies a concrete, GRPO-compatible training signal for mitigating strategic failure modes that arise directly from the language interface in multi-agent LLM agents. The explicit ablation demonstrating that a constant penalty vanishes while per-rollout computation does not is a strength, as is the release of code and datasets, which enables direct reproduction and extension. The work therefore supplies both an empirical method and supporting artifacts for the emerging area of strategic safety in language-model agents.","major_comments":[{"comment":"Abstract and §3 (objective definition): the three penalty terms (exploitability, collusion risk, externality) are described only at the level of natural-language names; no explicit functional forms, scaling constants, or estimation procedures from trajectories are given. Because the central claim that per-rollout computation produces a non-vanishing gradient under GRPO normalization rests on these definitions, their absence is load-bearing for both the method and the ablation.","section":"Abstract and §3"},{"comment":"Evaluation section (Kuhn Poker and cross-domain results): zero exploit-pool advantage, safety outperformance, and positive normalized relative advantage are reported without error bars, number of rollouts, or the precise procedure used to estimate exploitability from finite trajectories. These omissions directly affect the strength of the empirical claims that constitute the paper's primary evidence.","section":"Evaluation section"}],"minor_comments":[{"comment":"The abstract contains unreplaced LaTeX macros (\\sepo{}, \\href) that should be rendered in the camera-ready version.","section":"Abstract"}],"recommendation":"minor_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the positive assessment and for identifying specific areas where additional technical detail will improve clarity and reproducibility. We address each major comment below.","responses":[{"response":"We agree that the manuscript presents the penalty terms primarily through natural-language descriptions in the abstract and §3 without the explicit functional forms, scaling constants, or trajectory-based estimation procedures. This omission weakens the presentation of both the method and the ablation. In the revision we will expand §3 with the precise definitions (exploitability penalty as the estimated best-response value gap scaled by λ_e, collusion risk as a pairwise action-correlation term scaled by λ_c, externality as a third-party cost term scaled by λ_x) together with the Monte-Carlo estimation procedures used on sampled trajectories. The abstract will be updated to reference these definitions.","revision_made":"yes","referee_comment":"[Abstract and §3] Abstract and §3 (objective definition): the three penalty terms (exploitability, collusion risk, externality) are described only at the level of natural-language names; no explicit functional forms, scaling constants, or estimation procedures from trajectories are given. Because the central claim that per-rollout computation produces a non-vanishing gradient under GRPO normalization rests on these definitions, their absence is load-bearing for both the method and the ablation."},{"response":"We acknowledge that the evaluation section reports the key metrics without accompanying error bars, rollout counts, or a detailed description of how exploitability is estimated from finite trajectories. These details are required to substantiate the primary empirical claims. In the revised manuscript we will add standard-error bars computed across independent runs, state the number of evaluation rollouts per domain, and specify the estimation procedures (exact best-response computation via the game tree for Kuhn Poker; Monte-Carlo best-response approximation for the remaining domains).","revision_made":"yes","referee_comment":"[Evaluation section] Evaluation section (Kuhn Poker and cross-domain results): zero exploit-pool advantage, safety outperformance, and positive normalized relative advantage are reported without error bars, number of rollouts, or the precise procedure used to estimate exploitability from finite trajectories. These omissions directly affect the strength of the empirical claims that constitute the paper's primary evidence."}],"tokens_in":1543,"tokens_out":487,"duration_ms":22238,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is that this paper defines Safe Equilibrium Policy Optimization as an objective that adds explicit penalties for exploitability, collusion risk, and externality cost to the expected payoff, then trains language models with GRPO after SFT. They report zero exploit-pool advantage in Kuhn Poker and safety gains in four other domains while fixing SFT over-cooperation. The ablation directly tests that a shared constant penalty produces zero gradient under GRPO normalization while per-rollout computation does not.\n\nThe release of code and SFT datasets is useful, and the ablation gives concrete evidence that the mechanism is not trivial. The evaluation spans five domains with an explicit check on the computation choice, which addresses the gradient-vanishing concern head-on.\n\nThe soft spots are the missing details. The abstract supplies no equations for the penalty terms, no description of how exploitability is estimated from trajectories, no error bars, and no dataset or rollout sizes. Those gaps make it hard to judge robustness or sensitivity to the free weights on the three terms. The central claims rest on implementation choices that are not shown in the provided text.\n\nThis is for researchers working on safety interventions for language-model agents in negotiation, auctions, or repeated games. A reader who wants practical RL fixes and open artifacts will get value from the ablation and the released material.\n\nIt deserves peer review because the problem is scoped and relevant, the ablation is informative, and the artifacts allow checking, even though more method detail would help.","headline":"SEPO adds three per-rollout penalties to GRPO for LM agents and the ablation on why constants cancel is the clearest contribution.","tokens_in":2397,"tokens_out":370,"would_cite":false,"duration_ms":15903,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Safe Equilibrium Policy Optimization adds penalties for exploitability, collusion, and externalities to language-model agent training.","keywords":["safe equilibrium policy optimization","strategic agents","language models","exploitability","multi-agent reinforcement learning","group relative policy optimization","Kuhn poker","negotiation safety"],"falsifier":"A replication in which a shared constant penalty version of the same training loop produces equivalent safety gains and zero exploit advantage in Kuhn Poker would falsify the necessity of per-rollout computation.","tokens_in":2677,"feed_emoji":"🛡️","tokens_out":732,"duration_ms":21309,"temperature":0.7,"pith_summary":"The paper establishes that standard reinforcement learning on language models for strategic tasks ignores multi-agent structure, allowing exploitation of opponents, harmful collusion, and cost externalization through the language interface. SEPO augments the payoff objective with explicit penalties for these risks and uses the resulting reward in Group Relative Policy Optimization after supervised fine-tuning. Across Iterated Prisoner's Dilemma, repeated auctions, two negotiation settings, and Kuhn Poker, the approach produces agents with zero exploit-pool advantage in poker for both tested models and superior safety metrics in four domains. It also reverses the over-cooperative tendency introduced by supervised fine-tuning alone. Ablations show that per-rollout exploitability computation is required, as constant penalties cancel under advantage normalization.","feed_headline":"SEPO training yields zero exploit advantage in Kuhn Poker","feed_subtitle":"Penalizing exploitability and collusion in GRPO corrects unsafe behaviors from language-model agents across strategic domains.","key_machinery":"Safe Equilibrium Policy Optimization (SEPO), the objective that augments expected payoff with explicit penalties for exploitability, collusion risk, and externality cost before serving as the reward for Group Relative Policy Optimization.","core_discovery":"The paper claims that augmenting expected payoff with penalties for exploitability, collusion risk, and externality cost, then using this as the reward signal for GRPO on Gemma 4 E4B-it and Qwen 3.5-4B after SFT, yields agents that achieve zero exploit-pool advantage in Kuhn Poker for both models, outperform the base model on safety metrics in four of five domains, correct over-cooperative SFT behavior, and in negotiation produce the only positive-safety outcome with positive normalized relative advantage among tested configurations.","pith_inferences":["The per-rollout exploit computation step could be ported to other advantage-normalized RL methods to avoid gradient cancellation on safety terms.","Releasing the code and SFT datasets allows direct testing of whether the same penalties reduce harmful equilibria in additional language-mediated games.","If the approach scales, agents trained this way could be deployed in real multi-party negotiations or auctions without requiring post-hoc safety filters."],"forward_implications":["SEPO training reaches zero exploit-pool advantage in Kuhn Poker for both Gemma and Qwen models.","The method improves safety metrics over the base model in four of the five tested domains.","SEPO reverses the over-cooperative behavior that supervised fine-tuning alone introduces.","In negotiation domains, SEPO is the only configuration that reaches a positive-safety outcome with positive normalized relative advantage."],"fun_headline_variants":["SEPO achieves zero exploit advantage in Kuhn Poker","Zero exploit advantage via SEPO in Kuhn Poker","SEPO removes exploit advantage in Kuhn Poker","Exploit-free outcomes with SEPO in Kuhn Poker"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"That exploitability computed separately on each rollout from trajectories supplies a non-vanishing gradient once GRPO normalizes advantages, unlike a shared constant penalty that cancels as a control variate.","fun_headline_variants_meta":{"raw":{"variants":["SEPO achieves zero exploit advantage in Kuhn Poker","Zero exploit advantage via SEPO in Kuhn Poker","SEPO removes exploit advantage in Kuhn Poker","Exploit-free outcomes with SEPO in Kuhn Poker"]},"model":"grok-4.3","cost_usd":0.004864,"raw_usage":{"total_tokens":2428,"prompt_tokens":750,"num_sources_used":0,"completion_tokens":57,"cost_in_usd_ticks":48637000,"prompt_tokens_details":{"text_tokens":750,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1621,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":750,"tokens_out":57,"duration_ms":11268,"temperature":1.0,"reasoning_tokens":1621,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T20:53:09.971820+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A replication in which a shared constant penalty version of the same training loop produces equivalent safety gains and zero exploit advantage in Kuhn Poker would falsify the necessity of per-rollout computation.","supporting_citations":[],"review_version":1}