{"id":"69299ea7-eeab-4a45-aad0-0ca5141163d2","arxiv_id":"2506.16995","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"MPPO trains game agents with PPO on a mix of self-play and filtered demonstration trajectories, achieving proficiency near pure online PPO while keeping the demonstrator's action distribution closer than PPO does.","lead":"The paper introduces MPPO, a reinforcement learning algorithm that mixes a small amount of replayed demonstration data into PPO training to improve game agents' skill while keeping their original play style. A generalist reader might care because the method targets the common game-AI problem of balancing strength and behavioral diversity.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MPPO's monotonic-improvement proof does not cover the implemented loss: Algorithm 1 replays demonstrator actions, while Eq. (4) samples actions from the new policy, and the positive-return filter does not enforce Theorem 2's per-state advantage condition.","rationale":"The paper's headline claim is empirical, but its unique contribution includes a proof of monotonic improvement and style convergence. The reader's conditional verdict is appropriate. My stress-test focused on the link between the proof and the implementation. Eq. (4) is a surrogate for JΔ with actions drawn from the new policy at demonstration states. Algorithm 1's offline branch feeds recorded demonstrator actions into the same PPO ratio; because the behavior distribution is πT rather than πk, the ratio is not a valid importance weight for the intended expectation. Thus the BPPO/TRPO-based guarantee is not inherited. Additionally, filtering by positive total return does not enforce the per-state A>0 condition of Theorem 2. These are internal-consistency issues, not disagreements with consensus; the empirical findings are not disproved. The paper does provide code and extensive ablations, which support the empirical claims but not the proof. A faithful re-implementation of Eq. (4) would settle whether the mismatch matters. Consequently, the verdict remains CONDITIONAL.","tokens_in":13533,"tokens_out":5579,"duration_ms":58241,"concrete_test":"On Blackjack and Maze, run two learner configurations with identical β and seeds: (A) Algorithm 1's offline branch as written, using the recorded demonstrator action a_t in the PPO ratio; (B) a literal Eq. (4) offline branch that, for each stored state s_t, samples a_t_new ∼ πθ(·|s_t) and uses that action in the PPO objective, discarding the recorded action. Compare final win rates and Dpolicy. If (A) and (B) are statistically equivalent, the mismatch is immaterial; if they differ materially, the reported algorithm is not the one whose monotonic improvement is proved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim—that MPPO monotonically improves while preserving style—rests on Eq. (4) and Theorem 2, but the implemented algorithm optimizes a different objective. Eq. (4) defines the offline term as E_{s∼ρD, a∼π(·|s)}[Aπk(s,a)], sampling actions from the current/new policy at demonstration states. Algorithm 1 and the LfD actors instead replay the demonstrator's recorded actions a_t; the PPO ratio r=πθ(a_t|s_t)/π_k(a_t|s_t) is then an ordinary likelihood for a_t under πθ and π_k, not an importance weight correcting the mismatch between the replay distribution (a_t∼πT(·|s_t)) and the target distribution a∼πθ(·|s_t). The offline PPO term is therefore not an unbiased estimator of Eq. (4), and the appeal to BPPO/TRPO monotonic-improvement bounds does not transfer. Separately, Theorem 2 requires AπT(s_t,a_t)>0 for every state-action pair in the demonstration, but Section 4 filters only on positive total episode return, which does not imply per-step positive advantage; the proof also assumes a deterministic teacher (πT(a|s_t)=1_{a=a_t}) and ignores PPO clipping. These gaps mean the paper's formal guarantees do not apply to the reported method, leaving the proficiency/style-preservation results as heuristics.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Mixed Proximal Policy Optimization (MPPO), a learning-from-demonstration method that continues PPO training while replaying demonstration trajectories from suboptimal, style-specific bots. The authors claim a monotonic improvement guarantee and a monotonic decrease in style distance to the demonstrator, and support these claims with experiments in Blackjack, Maze Navigation, and MCR Mahjong, using win rates, success rates, and the proposed Dpolicy/Dtarget style metrics. They also report a compact demonstration-storage format and an ablation/comparison against PPO, DQfD, GAIL, SAIL, and PPOfD.","tokens_in":13828,"tokens_out":7214,"duration_ms":77243,"significance":"MPPO addresses a practically relevant problem, and the empirical comparison is substantial: three environments, repeated runs with error bars, ablations over demonstration ratio and advantage estimator, and comparisons against several learning-from-demonstration baselines. The code release, benchmark environments, and the simple Dpolicy metric are useful contributions. However, the theoretical guarantees are advertised as a main contribution, and as presented they do not cover the implemented algorithm. The paper's current value therefore rests on the empirical results, which are promising but would need the theoretical claims to be corrected or removed.","major_comments":[{"comment":"The offline term in Eq. (4) is E_{s∼ρ_D, a∼π(·|s)}[A_{π_k}(s,a)], sampling actions from the student policy at demonstration states. Algorithm 1's LfD actors instead replay the demonstrator's recorded actions a_t (Algorithm 1 line 6), and the PPO ratio r=π_θ(a_t|s_t)/π_k(a_t|s_t) is a likelihood ratio for the recorded action, not an importance weight that corrects the mismatch between the replay distribution a∼π_T(·|s) and the target distribution a∼π(·|s). The objective actually optimized is therefore not Eq. (4), and the BPPO/TRPO monotonic-improvement arguments invoked in Section 4 do not transfer to the implemented loss. This is a load-bearing gap because the monotonic improvement claim is stated as a formal contribution.","section":"§4, Eq. (4) and Algorithm 1"},{"comment":"Theorem 2 requires A_{π_T}(s_t,a_t)>0 for every (s_t,a_t) in each demonstration trajectory. Section 4 states that trajectories are filtered by positive total return, which does not imply per-step positive advantage; moreover, Algorithm 1 computes advantages with the student critic via GAE, not with the teacher policy π_T. Thus the theorem's precondition is neither enforced nor verifiable from the reported procedure. The 'All Data' ablation in Section 6 correctly notes that including all trajectories violates the theorem, but the positive-return filter used in the main experiments has the same logical gap.","section":"§4, Theorem 2 and positive-return filter"},{"comment":"The proof of Theorem 2 assumes clipping is not activated, assumes a deterministic teacher (π_T(a|s_t)=1_{a=a_t}), and analyzes a hand-specified update π'_S(a_t|s_t)=π_S(a_t|s_t)+α∇π_S(a_t|s_t)π_T(a_t|s_t)A_{π_T}(s_t,a_t) rather than the clipped PPO update in Eq. (2). As printed, the gradient step is not a normalized policy-gradient update, and no argument connects it to the loss used by Algorithm 1. Consequently the claimed monotonic decrease of D_policy under MPPO's offline component is not established for the implemented method.","section":"§4, Theorem 2 proof"}],"minor_comments":[{"comment":"Equation (1) should read E_{s∼ρ_π(·), a∼π'(·|s)}[A_π(s,a)]; the current notation E_{s∼π(·)} is undefined as written.","section":"§3, Eq. (1)"},{"comment":"In the displayed MPPO objective, the second clip term is printed as clip(r,1−ϵ,1+rM); it should presumably be clip(r,1−ϵ,1+ϵ).","section":"§4, PPO objective"},{"comment":"The theorem statement should explicitly state the deterministic-teacher assumption and should define D_policy consistently with Eq. (3), including the 1/2 factor and the expectation over states; the proof appears to analyze a single state s_t.","section":"§4, Theorem 2"},{"comment":"The text says 'as shown in Figure 2B' when reporting Maze D_policy results, but the results appear in Table 2B.","section":"§5.2"},{"comment":"The relationship between the sampling fraction β in Section 4 and the number of LfD actors used in Section 5 is not explicit; Section 5 says demonstration data accounts for approximately 5% via actor counts, while Section 4 defines β as a sampling fraction. Clarify whether these are the same mechanism.","section":"§4 and §5"},{"comment":"The D_policy computation for Mahjong demonstrators uses p(a|s)=1_{a=a_i} from 100 held-out trajectories, which treats the demonstrator as deterministic and may affect the measured distances; this limitation should be stated when interpreting Table 4B.","section":"§5.3"}],"recommendation":"major_revision","confidential_remarks":"The empirical sections are the strength of the paper; the theoretical claims in Section 4 should be substantially corrected or removed before publication. I see no issue with the use of Botzone data or the cited Mahjong environment paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about this paper: the empirical core is genuinely useful, and the theory is not. MPPO is a straightforward combination of PPO with a small replay fraction of positively-filtered demonstration trajectories, and the authors show across Blackjack, Maze, and Mahjong that it improves suboptimal bots while keeping their action distributions closer to the demonstrator than pure PPO. The Mahjong result, starting from suboptimal Botzone bots and beating the top-ranked bot, is the kind of thing that gets practitioners' attention. The paper ships code, runs ablations on the demo ratio and the advantage estimator, and compares against GAIL, SAIL, DQfD, and a PPOfD baseline. The storage-footprint reduction via recording seeds plus actions is a nice practical detail. The Dpolicy metric (total variation on action distributions) is simple and sensible.\n\nThe soft spot is exactly where the stress-test puts it: Theorem 2 does not cover the implemented algorithm. The filter keeps trajectories with positive total return, while the theorem needs AπT(s,a)>0 at every visited state-action pair. And Eq. (4) writes the offline term as an expectation over actions sampled from the new policy, but Algorithm 1 and the LfD actors replay the demonstrator's recorded actions. The PPO ratio is then just a likelihood ratio for those recorded actions, not an importance weight correcting the distribution mismatch, so the BPPO/TRPO monotonic-improvement argument does not transfer. The proof's algebra is also hard to follow and appears to assume a deterministic teacher and no clipping. These are not minor typos; they mean the paper's central theoretical claim is unsupported. That said, the empirical claims stand independently of the theory, and the ablations (All Data, TD(0) Adv) actually show behavior consistent with the idea that filtering matters.\n\nThe Mahjong win-rate comparison also mixes apples and oranges: demonstrator win rates come from Botzone historical Elo data, student win rates from 512 test games against the baseline. That should be aligned.\n\nBottom line: this is a useful empirical paper that overreaches in its theory. I'd send it to peer review because the empirical finding is worth scrutinizing and the authors are capable of fixing or removing the theory. The reader's CONDITIONAL verdict is right.\n\nRecommendation: engage with it, but insist the theory be either made correct or explicitly labeled as heuristic.","headline":"Useful empirical LfD method with a broken monotonic-improvement proof; the experiments are worth a look, the theory is not.","tokens_in":14379,"tokens_out":2324,"would_cite":false,"duration_ms":22120,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MPPO, a mixed online/offline PPO with positive-return demonstration filtering, improves suboptimal game agents to near-PPO proficiency while preserving their play styles.","keywords":["reinforcement learning","learning from demonstration","play style preservation","proximal policy optimization","game AI","mixed online-offline learning","Mahjong","policy distance"],"falsifier":"Compute, for each state-action pair in the filtered demonstration sets, the teacher's advantage $A_{\\pi_T}(s,a)$; if a large fraction are non-positive and $D_{\\mathrm{policy}}$ still decreases during training, the stated precondition is not the operative mechanism. A controlled comparison would train MPPO with per-state positive-advantage filtering versus episode-return filtering and check whether style retention differs.","tokens_in":1685,"feed_emoji":"🎮","tokens_out":1650,"duration_ms":96338,"temperature":0.7,"pith_summary":"Game agents face a tension: the methods that make them strong tend to erase what makes them distinct. This paper claims that a single training recipe, MPPO, dissolves that tension. MPPO takes a suboptimal agent that plays in a recognizable style, records its demonstrations, and trains a student with one PPO-style objective over a mixture of self-play samples and replayed demonstration samples. Across Blackjack, Maze, and MCR Mahjong, the paper reports that MPPO students reach win rates comparable to or better than pure online PPO while staying much closer to the demonstrator's action distribution, and one Mahjong student overtakes the strongest bot on the site's leaderboard. If true, this is a practical route to game bots that are both highly skilled and stylistically diverse.","feed_headline":"MPPO lifts suboptimal game bots to PPO-level skill, style intact","feed_subtitle":"A mixed online/offline PPO lets stylized, weaker bots reach near-PPO strength without flattening their identity.","key_machinery":"The central object is the mixed surrogate objective in Eq. (4), which estimates the performance difference $J_\\Delta(\\pi, \\pi_k)$ by sampling states from a mixture $\\beta\\rho_D + (1-\\beta)\\rho_{\\pi_k}$: the offline component draws states from the demonstration dataset and actions from the current policy, while the online component draws from the current policy's own state distribution. Each component is a clipped PPO surrogate, so the whole objective is a single loss applied uniformly to online and LfD actor samples. The style-preservation mechanism is implicit: demonstration trajectories are filtered to positive total return and replayed by LfD actors, biasing the empirical distribution of training samples toward the demonstrator; $D_{\\mathrm{policy}}$, a total-variation distance between action distributions, is used to verify that the student stays close to the teacher.","core_discovery":"The central claim is that proficiency and style are not competing objectives if demonstration data are treated as a soft prior rather than a hard imitation target. MPPO's objective is the PPO clipped surrogate applied to a mixture of state distributions: a fraction $\\beta$ of states sampled from the demonstration dataset and $1-\\beta$ from the policy's own rollouts. The offline component is justified by an offline policy-improvement result and the online component by the standard TRPO/PPO guarantees, giving the paper's claim of monotonic improvement; style preservation comes from filtering demonstrations to positive-return trajectories and replaying them through LfD actors, which shifts the empirical sample distribution toward the demonstrator without an explicit cloning loss. The paper measures the outcome with $D_{\\mathrm{policy}}$, a total-variation distance between action distributions, and, in Mahjong, with a distribution distance over winning patterns. In all three testbeds MPPO improves over the demonstrator and matches or beats PPO, with $D_{\\mathrm{policy}}$ values substantially lower than those of PPO.","pith_inferences":["A mechanism-separating experiment: train MPPO with demonstrations filtered by per-state positive advantage instead of positive episode return; if style retention is unchanged, the theorem's condition is not the operative ingredient.","The All Data ablation shows that dropping the filter eliminates style retention in every environment, so the filter threshold itself is an untested hyperparameter; sweeping it would map the style-versus-proficiency trade-off and likely find environment-specific optima, matching the observed ratio sensitivity.","The recipe is demonstrated on discrete-action games, but the same mixed-objective idea naturally extends to continuous-control settings where 'style' is a movement signature; that would require replacing $D_{\\mathrm{policy}}$ with a trajectory-level distance and testing whether replaying recorded actions still works when actions are continuous."],"forward_implications":["MPPO students surpass their demonstrators in all three environments and, in Mahjong, one MPPO agent overtakes the leaderboard's top-ranked bot despite starting from suboptimal demonstrations.","Style retention shows up at both the action level ($D_{\\mathrm{policy}}$) and strategy level (winning-pattern distribution in Mahjong), unlike PPO, which collapses pattern diversity.","The seed-plus-action replay format reduces offline dataset storage by at least 98% in the reported environments while still supporting full-episode GAE advantage estimation.","Only a small demonstration fraction, about 5% of incoming data, is needed for the effect, and the best fraction is environment-dependent, peaking at different ratios in Blackjack, Maze, and Mahjong.","Because MPPO's loss is the same as PPO's, adopting the method is an actor-side data-collection change rather than a new learner."],"supporting_citations":[{"why":"Supplies the performance-difference identity (Theorem 1) that MPPO's mixed objective is built to maximize.","marker":"Kakade and Langford 2002"},{"why":"Provides the TRPO monotonic-improvement guarantee that MPPO inherits for its online component.","marker":"Schulman et al. 2017a"},{"why":"Provides the BPPO offline improvement guarantee that MPPO attributes to the demonstration-state component.","marker":"Zhuang et al. 2023"},{"why":"Supplies the clipped PPO surrogate objective that MPPO uses as the unified loss for both sample sources.","marker":"Schulman et al. 2017b"},{"why":"Supplies GAE, the full-episode advantage estimator that MPPO applies to replayed demonstration trajectories.","marker":"Schulman et al. 2015"},{"why":"Prior work using a 2-Wasserstein play-style distance that MPPO cites as motivation for its own $D_{\\mathrm{policy}}$ metric.","marker":"Lin, Chiu, and Wu 2024"},{"why":"DQfD, the explicit-supervised-loss RLED baseline that MPPO contrasts with its implicit constraint.","marker":"Hester et al. 2017"},{"why":"GAIL, the adversarial IL baseline that MPPO compares against and argues cannot surpass the demonstrator.","marker":"Ho and Ermon 2016"}],"fun_headline_variants":["MPPO: weak bots gain PPO-level skill, keep their style","Style-preserving PPO: suboptimal bots match elite play","Mix online and offline: MPPO gives bots skill and style","No style sacrifice: MPPO boosts weak bots to elite tier"],"cache_read_input_tokens":16384,"weakest_assumption_plain":"The proof that MPPO preserves style assumes every action in the kept demonstrations is better than average for its state (positive advantage), but the implementation keeps only entire trajectories whose total return is positive, which does not guarantee that per-action condition.","fun_headline_variants_meta":{"raw":{"variants":["MPPO: weak bots gain PPO-level skill, keep their style","Style-preserving PPO: suboptimal bots match elite play","Mix online and offline: MPPO gives bots skill and style","No style sacrifice: MPPO boosts weak bots to elite tier"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001369,"raw_usage":{"total_tokens":5537,"prompt_tokens":919,"completion_tokens":4618,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":4543}},"tokens_in":535,"tokens_out":4618,"duration_ms":31866,"temperature":1.0,"reasoning_tokens":4543,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:15:17.411154+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, for each state-action pair in the filtered demonstration sets, the teacher's advantage $A_{\\pi_T}(s,a)$; if a large fraction are non-positive and $D_{\\mathrm{policy}}$ still decreases during training, the stated precondition is not the operative mechanism. A controlled comparison would train MPPO with per-state positive-advantage filtering versus episode-return filtering and check whether style retention differs.","supporting_citations":[{"cited_title":"Approximately optimal approximate reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Supplies the performance-difference identity (Theorem 1) that MPPO's mixed objective is built to maximize."},{"cited_title":"Behavior proximal policy optimization, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the BPPO offline improvement guarantee that MPPO attributes to the demonstration-state component."},{"cited_title":"Perfectdou: Dominating doudizhu with perfect information distillation, 2024","cited_arxiv_id":null,"evidence_quote":"Prior work using a 2-Wasserstein play-style distance that MPPO cites as motivation for its own $D_{\\mathrm{policy}}$ metric."}],"review_version":1}