{"id":"a01ee7b0-8235-471a-97eb-0c010d6e1346","arxiv_id":"2412.02016","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Exp3-IXrl combines reinforcement learning with the Exp3-IX equilibrium approximator, but its reported improvements are marginal and its key components are unspecified.","lead":"This paper introduces a hybrid algorithm that switches between a reinforcement learning agent and a game-theoretic equilibrium approximator to speed up learning in complex games. The authors test it in a cybersecurity simulation and in multi-armed bandits, but the reported gains are small or contradicted by their own table.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Exp3-IXrl's CCE guarantee is unsupported: the third-party observer switch with an unspecified normalization factor has no no-regret proof, and Section 5 reports no CCE-distance metric, so the central equilibrium-approximation claim is unverified.","rationale":"The paper's central contribution is the claim that Exp3-IXrl approximates a CCE in stochastic environments, but the algorithm as described cannot inherit Exp3-IX's no-regret guarantee. Exp3-IX's regret bound (Neu 2015) assumes the learner's own action sequence determines the losses it observes; a third-party observer that watches an RL policy receives losses for actions it did not choose, and no correction is specified. The 'normalization factor' (Section 4) is mentioned but never defined or analyzed, so the reader cannot verify that the observer effect is fixed. This is not merely a missing detail: without a valid regret bound, the output need not be a CCE, and the method reduces to an RL policy with an arbitrary bandit overlay. The empirical section compounds the problem: only cumulative reward is reported, never an equilibrium-quality measure, and the CC2 table entry appears to contradict the text's 'comparable performance' claim (Table 1: -3.86 +/- 1.50 vs. -2.94 +/- 1.41). Because the central claim rests on an unproven and likely false theoretical assumption, and the provided evidence does not support it, the appropriate verdict is REJECT.","tokens_in":4724,"tokens_out":4947,"duration_ms":46102,"concrete_test":"Run Exp3-IXrl on a small finite stochastic game (e.g., a 2-player, 2-action repeated game) where the exact CCE is computable by linear programming. Train with the same threshold procedure and then compute the swap regret or CCE gap of the final mixed strategy. Separately, recompute the regret of the Exp3-IX component under the off-policy loss sequence with and without the proposed normalization factor. If the CCE gap is not below a pre-specified epsilon, or if the off-policy regret is not sublinear in the number of rounds, the CCE approximation claim fails.","verdict_should_be":"REJECT","load_bearing_attack":"The most load-bearing concern is that the paper never establishes that Exp3-IXrl outputs a coarse correlated equilibrium. Section 4 defines Exp3-IXrl as a 'third-party observer' that follows an RL teacher until a 'certainty threshold' and then switches to the CCE policy, with an unspecified 'normalization factor' claimed to fix the observer effect. The no-regret guarantee of Exp3-IX (Neu, 2015) applies to a learner that selects actions and observes their losses; during the observation phase, actions are chosen by the RL teacher, so the observed losses are off-policy and do not correspond to Exp3-IX's own recommendations. No derivation, no importance-weighting scheme, and no regret bound is provided for this observer regime. Moreover, Section 5 reports only average cumulative reward over 30 steps; no swap regret, external regret, or CCE distance is measured. In fact, Table 1 shows Exp3-IXrl's CC2 reward (-3.86 +/- 1.50) is worse than the PPO baseline (-2.94 +/- 1.41), contradicting the text's claim of comparable performance. Without an equilibrium-quality metric, the central claim that the method approximates a CCE is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Exp3-IXrl, an algorithm that combines an RL teacher policy with the Exp3-IX bandit algorithm to approximate a coarse correlated equilibrium (CCE) in stochastic environments. The RL teacher selects actions during an observation phase; once a so-called certainty threshold is reached, the algorithm switches to a CCE policy computed from Exp3-IX. The authors claim that this separation accelerates training and enables equilibrium approximation in complex environments such as CybORG CC2 and multi-armed bandits. Experiments report average cumulative reward over 30 steps after 10,000 training episodes, comparing against PPO in CC2 and against classical bandit algorithms in MAB settings.","tokens_in":5103,"tokens_out":4325,"duration_ms":37828,"significance":"The idea of decoupling the RL agent's exploration from the equilibrium computation is an interesting direction that could be relevant to the growing literature on combining game-theoretic and RL methods. If the method actually produced a CCE with a verifiable no-regret guarantee, it would be a meaningful contribution to equilibrium approximation in large stochastic games. The paper also correctly identifies a real gap: existing CCE algorithms often struggle in complex environments where RL-based policies are fast but lack game-theoretic guarantees. However, the current manuscript does not substantiate either the theoretical or the empirical claims: the central equilibrium-quality property is never defined or measured, the algorithm is underspecified, and the main experimental result in CC2 is contradicted by the paper's own table.","major_comments":[{"comment":"The claim that the algorithm 'significantly outperforms its classical RL and CCE counterparts' is directly contradicted by the reported CC2 results. Table 1 shows Exp3-IXrl achieving -3.86 +/- 1.50 cumulative reward over 30 steps, while the PPO (CardiffUni) baseline achieves -2.94 +/- 1.41. Since higher rewards are better in this environment, Exp3-IXrl is worse than the baseline, not comparable or better. The abstract's promise of 'improved performance' and 'significantly outperforms' is therefore unsupported by the paper's own data.","section":"Section 5.3, Table 1"},{"comment":"The third-party observer mechanism breaks the applicability of Exp3-IX's no-regret guarantee. During the observation phase, actions are chosen by the RL teacher, not by Exp3-IX, so the losses observed are not those incurred by Exp3-IX's own recommendations. The paper states that 'we introduce a normalization factor to alleviate the third-party observer effect' but never defines this factor or provides any derivation of a regret bound. No swap regret, external regret, or distance-to-CCE metric is reported in Section 5. Consequently, the central claim that Exp3-IXrl approximates a coarse correlated equilibrium is not established.","section":"Section 4"},{"comment":"The experimental evaluation never measures equilibrium quality. The only metric reported is average cumulative reward over 30 steps. Since the paper's contribution is equilibrium approximation, the reader cannot verify that the output is any closer to a CCE than the RL baseline alone. A regret measurement or a comparison of the final joint policy against a known equilibrium would be necessary to support the claim of CCE approximation; no such metric appears anywhere in the manuscript.","section":"Section 5.2, Section 5.3"},{"comment":"The certainty threshold is an environment-specific free parameter whose value (2750 for CC2, 2000 for MAB) is not derived from any principle or stated selection procedure. Given that Fig. 2 plots reward as a function of the threshold, the chosen value appears to be selected post hoc to match the PPO baseline. This makes the reported 'faster convergence' result a fitted outcome rather than a predictive one. Without a principled way to set the threshold, the method is not reproducible and the comparison to PPO is not meaningful.","section":"Section 4, Section 5.3, Fig. 2"}],"minor_comments":[{"comment":"The reference for PPAD-completeness of Nash equilibrium in two-player games is given as [2] 'Three-player games are hard', which is not the correct classic reference; the relevant result is Daskalakis, Goldberg, and Papadimitriou (2009). The abbreviation '2p0s' should also be spelled out.","section":"Section 1"},{"comment":"The regret definitions contain minor notational inconsistencies: the loss ℓ_{t,k} is indexed with k on the left but with i in the summation, and the CCE definition uses cost c_i while the game definition uses rewards R_i; the sign convention should be clarified.","section":"Section 2"},{"comment":"The table is difficult to parse because of the unaligned columns and repeated baseline names; please reformat with clear column headers for Exp3, Exp3-IX, RL teacher, and Exp3-IXrl, and with one row per environment/teacher combination.","section":"Table 1"},{"comment":"The paper would benefit from a formal pseudocode block for Exp3-IXrl, because the textual description alone does not define the certainty measure, the normalization factor, or the exact switching rule.","section":"Section 4, Fig. 1"},{"comment":"Reference [7] has a typo: 'Eqilibrium' should be 'Equilibrium'. Additionally, the paper does not state whether code is available, which would be useful for reproducibility.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper appears to be an early-stage draft: the core algorithm is described informally, the main theoretical term (normalization factor) is undefined, and the headline empirical claim is contradicted by the paper's own table. I see no quick local fix that would bring the manuscript to a publishable state without additional theoretical analysis and new experiments that measure regret or CCE distance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has a genuinely interesting idea — use an RL agent as a third-party explorer for Exp3-IX and switch to the bandit's policy once a certainty threshold is hit — but it never checks whether the output is actually a coarse correlated equilibrium, and its flagship experiment in CybORG shows worse performance than the PPO baseline while the text claims otherwise. This is not ready for publication.\n\nWhat's new: the specific hybrid architecture, which separates action selection from CCE computation, is not in the cited literature. That separation is a sensible way to leverage RL's exploration speed while keeping the bandit's no-regret machinery. The MAB results are mostly competitive, and the CybORG domain is a reasonable stress test. The paper is also honest enough to raise the third-party observer question in Section 4, and it mentions a normalization factor as a fix.\n\nThe soft spots are serious. First, the no-regret guarantee of Exp3-IX does not transfer to the observer regime. During the teacher phase, actions are chosen by the RL policy, so Exp3-IX observes off-policy losses. The normalization factor is never defined, and no importance weighting or alternative derivation is provided. Second, Section 5 measures only cumulative reward. There is no swap regret, external regret, or CCE-distance metric anywhere. So the central claim that this method approximates a CCE is unverified. Third, Table 1 shows CC2 reward of -3.86 ± 1.50 for Exp3-IXrl versus -2.94 ± 1.41 for the PPO baseline. The text says 'comparable performance' and the abstract implies it outperforms. That is a direct contradiction. Fourth, the certainty threshold of 2750 looks post-hoc: the paper shows a figure and picks the threshold that matches PPO. That is fitting, not prediction.\n\nWho is this for? Someone looking for heuristic tricks to speed up CCE approximation in large stochastic games might find the architecture worth trying, but the paper does not establish that it works. The idea could be the seed of a solid workshop paper if the authors supply a real regret analysis for the observer regime and replace the reward-only evaluation with an equilibrium-quality metric. In its current form, I would not cite it or send it to a serious referee.","headline":"A promising heuristic that never verifies its equilibrium claim, and whose own data contradicts its conclusions.","tokens_in":5523,"tokens_out":5491,"would_cite":false,"duration_ms":48404,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Exp3-IXrl combines a reinforcement learning agent's actions with Exp3-IX's no-regret bandit updates and, after a certainty threshold, follows the resulting coarse correlated equilibrium policy, a blend the paper claims reaches CybORG…","keywords":["coarse correlated equilibrium","Exp3-IX","no-regret learning","reinforcement learning","multi-armed bandits","cybersecurity network environment","CybORG","certainty threshold"],"falsifier":"Run Exp3-IXrl on a ten-armed adversarial bandit, record the action sequence actually chosen, compute the empirical regret against the best fixed arm, and compare the final policy's joint action distribution to the CCE inequalities; if regret fails to grow sublinearly or any unilateral deviation improves expected cost at the reported certainty threshold, the paper's central claim is false.","tokens_in":4569,"feed_emoji":"🎯","tokens_out":8780,"duration_ms":79502,"temperature":0.7,"pith_summary":"Exp3-IXrl is a proposed hybrid that separates action selection from equilibrium computation: a reinforcement learning agent chooses actions during training, Exp3-IX — an adversarial bandit algorithm with a no-regret guarantee — records the losses as a third-party observer, and once a certainty threshold is reached the agent switches to the coarse correlated equilibrium (CCE) policy the bandit has computed. The paper's central claim is that this preserves the theoretical strength of CCE approximation while inheriting RL's speed in large stochastic environments, and that the resulting agent outperforms both pure RL baselines and plain Exp3/Exp3-IX in ten-armed bandit tests and in the adversarial CybORG CC2 cybersecurity environment. A sympathetic reader would care because equilibrium approximation algorithms have been confined to small or bandit settings, whereas this design points toward a practical way to bring no-regret equilibrium concepts into realistic multi-step games. The reported headline result is matching the CybORG CC2 PPO champion's cumulative reward in 10,000 simulation episodes, a tenth of the prior winning submission's training budget.","feed_headline":"RL-guided bandit reaches coarse equilibrium faster in cyber games","feed_subtitle":"Reinforcement learning drives exploration while a bandit computes the equilibrium, cutting CybORG training to one tenth.","key_machinery":"The load-bearing mechanism is Exp3-IX, a high-probability no-regret variant of the Exp3 adversarial bandit algorithm, placed in a new role: third-party observer rather than active player. At each timestep the RL agent selects the action, while Exp3-IX updates its weights from the observed losses; a certainty threshold, tied to how often a state has been visited, controls the handoff to the CCE policy. The paper also introduces a normalization factor meant to correct for the effect of the bandit no longer being the one choosing actions, so that the no-regret convergence story is supposed to survive the separation. Together these pieces let the RL teacher drive exploration during training while the final policy is the equilibrium object.","core_discovery":"On the paper's own terms, the discovery is that an equilibrium approximation algorithm can be made to work in a complex stochastic environment if the equilibrium learner is allowed to observe rather than act during training. Exp3-IXrl uses the RL agent as an exploration bias: the more often a state is visited, the more certainty the CCE approximation has for that state, and after the certainty threshold is crossed the bandit's policy takes over. The authors report that this scheme 'significantly outperforms its classical RL and CCE counterparts' in the bandit settings (with one stochastic gradient-bandit exception) and reaches performance comparable to the CardiffUni PPO agent in CybORG CC2 in a tenth of the episodes. They further claim this expands the application of equilibrium approximation algorithms to new environments.","pith_inferences":["A direct empirical regret check — comparing the actions actually taken by Exp3-IXrl against the best fixed arm in hindsight — would settle whether the CCE claim holds; the paper reports cumulative reward only, so the no-regret part of the story is currently unverified.","Because the certainty threshold is a fixed count per state, an obvious extension is to make it responsive to reward variance or prediction error; the paper itself flags adaptive certainty as future work.","If the normalization factor does not fully restore the no-regret guarantee, the CybORG gains could be attributable to the RL exploration bias alone, making Exp3-IXrl an RL algorithm with a bandit-shaped fallback rather than a true equilibrium approximation.","The same decoupling should transfer to multi-agent settings, where several Exp3-IXrl agents could jointly approach a CCE in environments that are currently too large for direct equilibrium computation."],"forward_implications":["Equilibrium approximation algorithms can be deployed in large stochastic multi-step environments, not just small or bandit games.","A competitive equilibrium-guided policy in CybORG CC2 can be trained in roughly 10,000 episodes instead of the 100,000 episodes used by the earlier winning submission.","Because the action selector and the equilibrium computer are decoupled, the same Exp3-IXrl wrapper can be paired with different RL teachers, from epsilon-greedy to UCB to gradient bandits.","In ten-armed bandit benchmarks, the hybrid's 30-step cumulative reward meets or exceeds plain Exp3, Exp3-IX, and the RL baselines in every configuration except stochastic gradient bandit.","If the CCE guarantee is intact after the observer switch, users get a game-theoretically stronger solution concept than a pure learned policy without paying RL's usual convergence costs."],"supporting_citations":[{"why":"Defines Exp3, the base no-regret adversarial bandit algorithm that Exp3-IXrl extends.","marker":"[5]"},{"why":"Defines Exp3-IX, the high-probability no-regret variant whose certainty-threshold observer role drives the hybrid.","marker":"[6]"},{"why":"Defines the CybORG simulator used for the CC2 experiments.","marker":"[19]"},{"why":"Defines the Cage Challenge 2 scenario and the 100,000-episode baseline budget the paper compares against.","marker":"[20]"},{"why":"Supplies the convergence proof for actor-critic methods applied to PPO, supporting the CardiffUni baseline's standing as a strong RL reference.","marker":"[18]"},{"why":"Defines PPO, the hierarchical algorithm underlying the CardiffUni agent used as the CC2 baseline.","marker":"[16]"},{"why":"Supplies the epsilon-greedy baseline and teacher in the bandit experiments.","marker":"[21]"},{"why":"Supplies the UCB baseline and teacher in the bandit experiments.","marker":"[22]"},{"why":"Supplies the gradient bandit baseline and teacher in the bandit experiments.","marker":"[23]"}],"fun_headline_variants":["RL guides bandit to equilibrium, cutting CybORG training to 10%","Equilibrium learner observes RL agent, then takes control in CybORG","Explore Reinforced: RL-driven exploration accelerates equilibrium approximation","RL and equilibrium combo achieves CybORG results in one-tenth episodes","Equilibrium approximation uses RL exploration to reach CybORG parity quickly"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The output is a true coarse correlated equilibrium: the algorithm assumes Exp3-IX's no-regret guarantee survives its change from active player to passive observer after the certainty threshold, and no regret or equilibrium-distance measurement is reported to check that assumption.","fun_headline_variants_meta":{"raw":{"variants":["RL guides bandit to equilibrium, cutting CybORG training to 10%","Equilibrium learner observes RL agent, then takes control in CybORG","Explore Reinforced: RL-driven exploration accelerates equilibrium approximation","RL and equilibrium combo achieves CybORG results in one-tenth episodes","Equilibrium approximation uses RL exploration to reach CybORG parity quickly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001395,"raw_usage":{"total_tokens":5574,"prompt_tokens":809,"completion_tokens":4765,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":425,"completion_tokens_details":{"reasoning_tokens":4672}},"tokens_in":425,"tokens_out":4765,"duration_ms":30696,"temperature":1.0,"reasoning_tokens":4672,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:54:57.780808+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Exp3-IXrl on a ten-armed adversarial bandit, record the action sequence actually chosen, compute the empirical regret against the best fixed arm, and compare the final policy's joint action distribution to the CCE inequalities; if regret fails to grow sublinearly or any unilateral deviation improves expected cost at the reported certainty threshold, the paper's central claim is false.","supporting_citations":[{"cited_title":"Explore no more: Improved high-probability regret bounds for non-stochastic bandits","cited_arxiv_id":null,"evidence_quote":"Defines Exp3-IX, the high-probability no-regret variant whose certainty-threshold observer role drives the hybrid."},{"cited_title":"Convergence Proof for Actor-Critic Methods Applied to PPO and RUDDER, pages 105–130","cited_arxiv_id":null,"evidence_quote":"Supplies the convergence proof for actor-critic methods applied to PPO, supporting the CardiffUni baseline's standing as a strong RL reference."},{"cited_title":"A Bradford Book, 2018","cited_arxiv_id":null,"evidence_quote":"Supplies the epsilon-greedy baseline and teacher in the bandit experiments."},{"cited_title":"Flaxman, Adam Tauman Kalai, and H","cited_arxiv_id":null,"evidence_quote":"Supplies the gradient bandit baseline and teacher in the bandit experiments."}],"review_version":1}