{"id":"883cea9b-a1b6-4662-b471-023de19d611a","arxiv_id":"2507.16473","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A variational option-critic algorithm with latent option embeddings and an implicit chain-of-thought cold-start is presented; the central optimality-preservation proof has a gap and some reported benchmark wins are inaccurate.","lead":"The paper proposes VMOC, an off-policy hierarchical reinforcement learning algorithm that learns reusable options as compact latent vectors, plus a cold-start procedure that distills explicit chain-of-thought demonstrations into those vectors. It also claims a mathematical guarantee that abstract-space learning preserves optimality, but the proof has a gap and several headline results are overstated.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 11's induction is invalid as written: it abstracts the next-state action with the current-state map g_{e_t} and replaces the abstract action's option component with f_bundle(o), neither of which is justified by Definition 8, so the central optimality-preservation guarantee is unproven.","rationale":"The strongest claim of the paper is the optimal value equivalence in Theorem 11, which undergirds the abstract's statement that learning in the simplified latent space preserves optimality. The reader identified the same step as the weakest assumption: the induction incorrectly uses the current-state abstraction map for the next-state action. My analysis sharpens this: beyond the index error, line (20) also assumes the abstract action's option component equals f_bundle(o), an additional structural property not present in Definition 8. Because both moves are needed to prove the Bellman operator commutes with the homomorphism, the central theorem is not proven in the manuscript. The theorem is likely fixable, but the paper's stated claim of a 'rigorous foundation' is not supported, and the additional empirical contradiction in the GSM-HARD claim means the paper should not be accepted in its current form. I therefore keep the reader's REJECT verdict, while noting that the theoretical result may be repairable in a major revision.","tokens_in":26802,"tokens_out":19272,"duration_ms":182881,"concrete_test":"Independently re-derive the induction in Theorem 11: apply the induction hypothesis at state e_{t+1} with map g_{e_{t+1}}, and replace line (20) with a supremum over the full abstract action space A×O (no f_bundle in the action). Check whether the Bellman recursion closes under Definition 8 alone. If it closes, Theorem 11 holds and the manuscript proof is corrigible; if not, the central claim is false as stated. A concrete mechanical check: formalize the induction in a proof assistant for a finite-state, finite-action HiT-MDP with state-dependent g_e and verify whether Q_m=Qbar_m for all m.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that learning in the abstract HiT-MDP preserves optimality (Theorem 11). The proof of this theorem is an induction on equations (16)-(22). Step (19) applies the induction hypothesis to the next-state action pair (a_{t+1},o_{t+1}) using g_{e_t}, the abstraction map attached to the current state e_t=(s_t,o_{t-1}). The hypothesis only provides Q_{m-1}(e,a,o)=Qbar_{m-1}(f(e),g_e(a,o)) with the map attached to the same state e; for the next step the relevant state is e_{t+1}=(s_{t+1},o_t). Definition 8 imposes no relation between g_{e_t} and g_{e_{t+1}}, so the switch is unjustified. Step (20) further replaces the abstract action by a pair (\\bar a_{t+1}, f_bundle(o_{t+1})), silently assuming the option component of g_e(a,o) is always f_bundle(o), which is not in Definition 8 (f is only required to be a bundle map; g_e is an arbitrary surjective Borel map). These steps are essential for closing the Bellman recursion, so Q*=Qbar* is not established. The theorem may be salvageable by a corrected proof, but the manuscript's rigorous-foundation claim is unsupported as written. A separate empirical misstatement (text claims best GSM-HARD 15.6 while Table 1 shows CoT-SFT 15.8) reinforces the need for full revision, but the proof failure is the load-bearing issue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VMOC, an off-policy maximum-entropy option-learning algorithm for HiT-MDPs, together with a cold-start procedure that distills explicit chain-of-thought demonstrations into discrete latent option embeddings for LLM reasoning. The authors also develop a theory of continuous HiT-MDP homomorphisms using vector bundles and claim that learning in the resulting abstract option space preserves optimal value functions (Theorem 11), that lifted policies exist and preserve value (Definition 13, Proposition 14, Theorem 15), and that maximizing the abstract ELBO is a valid surrogate for the original variational objective (Theorem 16). The paper reports experiments on MuJoCo locomotion and on mathematical/logical reasoning benchmarks.","tokens_in":27151,"tokens_out":5109,"duration_ms":59875,"significance":"If the central theoretical claims were correct, the paper would make a valuable contribution: it would give a principled account of when option-based abstractions preserve optimality in continuous state-action spaces and would connect that theory to an off-policy algorithm and to latent reasoning in LLMs. The paper also has practical strengths: it proposes a concrete off-policy algorithm, provides implementation details, ships code in the supplemental material, and evaluates against several baselines in both control and language domains. However, the main theoretical guarantee—optimal value equivalence under continuous HiT-MDP homomorphisms—is not established as written, and the convergence claim for neural-network VMOC rests on an unsupported assertion about stochastic gradient descent. The manuscript therefore does not currently deliver the rigorous foundation it advertises.","major_comments":[{"comment":"The induction step applies the induction hypothesis to the next-state action pair (a_{t+1}, o_{t+1}) using g_{e_t}, the abstraction map attached to the current state e_t=(s_t,o_{t-1}), rather than g_{e_{t+1}}. Definition 8 only supplies, for each e, a surjective Borel map g_e; it imposes no relation between g_{e_t} and g_{e_{t+1}}. Therefore the equality Q_m(e,a,o)=Qbar_m(f(e),g_e(a,o)) is not extended to the next state, and the Bellman recursion in equations (16)-(22) is unjustified. This is the central load-bearing step of the paper, and it invalidates the claim that Q^*(e,a,o)=Qbar^*(f(e),g_e(a,o)). A corrected proof would need an additional consistency condition on the family {g_e}, such as an equivariance relation along transitions; that condition is absent from Definition 8.","section":"§5.1, Theorem 11, proof step (19)"},{"comment":"Even if the map g_{e_t} could be applied at e_{t+1}, the transition from a supremum over g_{e_t}(a_{t+1},o_{t+1}) to a supremum over (\\bar a_{t+1}, f_bundle(o_{t+1})) is not justified. Definition 8 requires g_e to be surjective as a map A×O → Abar×Obar, but it does not require its option component to equal f_bundle(o), nor does it require f_bundle to be surjective onto the abstract option space. The change of variables in equation (21) also conflates the abstract state f(e_{t+1}) with e_{t+1} in the notation of Qbar_m. These steps are essential to closing the induction, so the proof of Theorem 11 does not go through.","section":"§5.1, Theorem 11, proof step (20)"},{"comment":"The proof of Theorem 3 asserts that 'using stochastic gradient descent (SGD) to optimize the parameters guarantees that the ELBO increases' and then concludes convergence of the sequence {q_k}. This is false in general for neural-network policies: the ELBO is nonconvex, stochastic gradients do not guarantee monotone increase, and convergence to a global or even local optimum is not assured without substantial additional assumptions. Since the theorem is used to justify the convergence of VMOC in Section 3.2, the paper's convergence claim for the practical algorithm is unsupported as stated.","section":"§A.4, Theorem 3"},{"comment":"Theorem 16 establishes the inequality L(q^↑) ≥ Lbar(q). The subsequent text concludes that 'improving the policy in the abstract space guarantees an improvement in the variational objective of the original problem.' That conclusion does not follow: the gap between L(q^↑) and Lbar(q) is the expected conditional entropy term in equation (37), and this gap can change when q changes. An increase in Lbar(q) therefore does not imply an increase in L(q^↑). Additionally, the proof assumes that the regularizer f is invariant under the homomorphism, but Definition 8 only states reward invariance; the manuscript does not show that the mutual-information regularizer used in Section 3 has this property.","section":"§6.2, Theorem 16 and following paragraph"}],"minor_comments":[{"comment":"The text states that VMOC-SFT achieves the best score on GSM-HARD with 15.6, but Table 1 lists CoT-SFT at 15.8, which is higher. This empirical claim should be corrected.","section":"§7.2.4, Table 1"},{"comment":"The transition equivariance condition is written as τ_{g_e(a,o)}(B|f(e)) = τ_{(a,o)}(f^{-1}(B)|s), but the argument on the right should presumably be e (or the corresponding state component) rather than s. Please clarify the notation.","section":"Definition 8"},{"comment":"The same symbol Q_m is used for both the original and the abstract value sequence, which makes it difficult to track whether an equality is between corresponding quantities. Using Qbar_m for the abstract MDP would improve readability and prevent apparent circularity.","section":"§5.1, equations (16)-(22)"},{"comment":"The proof of Theorem 3 refers to 'assuming the continuity and differentiability of q with respect to its parameters' but does not state which function class is considered; this should be made precise before convergence claims are made.","section":"§A.4"}],"recommendation":"reject","confidential_remarks":"The central theoretical contribution is not established: the proof of Theorem 11 has a genuine gap in the induction step, and Theorem 3 relies on an unjustified convergence assumption for SGD on neural-network ELBOs. These are load-bearing issues that the current manuscript cannot resolve by local editing; a substantial reformulation of the homomorphism conditions and of the convergence claims would be required. The paper also relies heavily on the authors' own prior work on HiT-MDP equivalence without independent verification, but that is secondary to the proof gap. If the authors can supply a corrected theorem with a suitable consistency condition on g_e and a defensible treatment of the variational convergence, the underlying idea may be worth reconsidering."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read it. The central theoretical guarantee—Theorem 11, that learning in the abstract HiT-MDP preserves optimality—is not established as written: the induction step applies the current-state abstraction map g_{e_t} to the next-state action, and Definition 8 imposes no consistency across states. And the paper's own Table 1 contradicts the text's claim of a best GSM-HARD score: the table shows CoT-SFT at 15.8, VMOC-SFT at 15.6. The load-bearing result fails and a headline empirical claim is misreported.\n\nThe paper is not without merit. The VMOC algorithm is a genuine combination: off-policy maximum-entropy option learning on HiT-MDPs with option embeddings, extending the on-policy MOPG. The continuous HiT-MDP homomorphism via vector bundles is a new extension of Panangaden et al., and the cold-start idea—distilling explicit CoT into discrete latent options for implicit reasoning—is worth exploring. The MuJoCo results look plausible, with code available.\n\nThe soft spots beyond Theorem 11: Theorem 3 claims SGD monotonically increases the ELBO and converges for neural policies; that is an assumption, not a proved statement. Theorem 16's ELBO lower-bound argument has a step where the conditional entropy of the concrete action given its abstraction is treated as non-negative extra; that is fine, but it relies on the lifted-policy definition which itself depends on the homomorphism properties. The language experiments do not support the broad claim: VMOC-SFT underperforms CoT-SFT on four of six datasets, and the only claimed win is the one the table refutes.\n\nProportionate verdict: the paper's central argument fails, but the ideas are salvageable. The proof might be repaired with a consistency condition on the abstraction maps, or the claim weakened. The empirical story needs honest reporting.\n\nThis paper is for a reader interested in hierarchical RL and latent reasoning who wants the novel combination, not for someone looking for a rigorous foundation. If I were the editor, I'd send it to a serious referee—the flaws are technical and fixable, and the novelty is real—but the current version should not be accepted without major revision.","headline":"A novel combination with a broken load-bearing proof: Theorem 11's induction misuses the state-indexed abstraction map, and the paper's own Table 1 contradicts its headline GSM-HARD claim.","tokens_in":27746,"tokens_out":2002,"would_cite":false,"duration_ms":22008,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that learning a policy in the abstract latent option space of a continuous hierarchical MDP preserves the optimal value function of the original problem.","keywords":["hierarchical reinforcement learning","options","variational inference","MDP homomorphisms","temporal abstraction","latent reasoning","chain-of-thought","maximum entropy reinforcement learning"],"falsifier":"Build a two-state HiT-MDP with overlapping action-option sets and choose surjective abstraction maps $g_{e_1}$ and $g_{e_2}$ that disagree on a shared action. If $Q^*(e_1,a,o) \\neq \\bar{Q}^*(f(e_1), g_{e_1}(a,o))$ while Definition 8 is satisfied exactly as written, the claimed equality is false.","tokens_in":26504,"feed_emoji":"🧠","tokens_out":7884,"duration_ms":75491,"temperature":0.7,"pith_summary":"This paper tries to establish that an agent can solve the original hierarchical decision problem by learning in a simplified abstract space of latent options, without any loss of optimality, provided the abstraction is a continuous homomorphism between Hidden Temporal Markov Decision Processes. It introduces VMOC, an off-policy variational option-learning algorithm that represents options as embeddings and maximizes an entropy-augmented objective. The proof centers on Theorem 11, which states that $Q^*(e,a,o) = \\bar{Q}^*(f(e), g_e(a,o))$ for a continuous HiT-MDP homomorphism $h=(f,g_e)$, and on Theorem 16, which states that the abstract evidence lower bound is a lower bound of the original evidence lower bound under the lifted policy. The paper also argues that explicit chain-of-thought demonstrations can be distilled into this latent option space during a cold-start supervised fine-tuning phase, giving LLM-style reasoning a structured initialization.","feed_headline":"Optimality survives abstraction in option-based RL","feed_subtitle":"Theorem 11 shows Q-values are equal under HiT-MDP homomorphisms, so latent option learning keeps the original solution.","key_machinery":"The object that carries the argument is the continuous HiT-MDP homomorphism $h=(f,g_e)$: $f$ is a vector-bundle map that abstracts the joint state-option space, and $g_e$ is a surjective Borel map, chosen per state $e$, that abstracts the augmented action-option space. Its two defining properties, invariance of reward and equivariance of transitions, are what allow the proof to push optimal $Q$-functions from the original MDP to the abstract MDP. The second mechanism is the variational ELBO with option embeddings: options are rows of a learnable embedding matrix, and optimizing the ELBO produces maximum-entropy exploration because the entropy terms $H[\\pi^A]$ and $H[\\pi^O]$ appear naturally in the objective. Together these two mechanisms let VMOC train in a simplified space while retaining a formal link to the original problem.","core_discovery":"On its own terms, the paper's central discovery is that the value-equivalence and policy-lifting results known for continuous MDP homomorphisms extend to the option-based HiT-MDP setting, and that this extension justifies variational learning in the abstract space. The authors define a continuous HiT-MDP homomorphism as a bundle map $f$ on the state-option space together with a state-dependent surjective Borel map $g_e$ on action-option pairs, satisfying reward invariance and transition equivariance. They then prove by induction that the optimal action-value function is unchanged by the abstraction, so any policy that is optimal in the abstract latent space corresponds to an optimal policy in the original problem. They further prove that the evidence lower bound optimized by VMOC on the abstract MDP is no larger than the evidence lower bound of the original MDP under the lifted policy, which means abstract-space variational training is a valid surrogate for the original objective.","pith_inferences":["Editorial inference: the proof's reliance on a state-dependent abstraction map suggests a testable consistency condition: if $g_e$ is required to agree across states that share actions, Theorem 11's induction becomes robust, and checking whether natural learned abstractions satisfy this condition would separate the theorem's scope from its current definition.","Editorial inference: the ELBO inequality $L(q^{\\uparrow}) \\ge \\bar{L}(\\bar{q})$ implies a practical monitoring statistic: during VMOC training, tracking the difference between the original and abstract ELBOs could reveal when the learned abstraction drifts outside the homomorphism class.","Editorial inference: the cold-start procedure should be compared against a continuous-latent VAE with the same compute; that comparison would isolate whether discrete option structure, rather than latent compression alone, drives the gains on logical reasoning tasks."],"forward_implications":["If Theorem 11 holds, a policy learned in the abstract latent option space can be lifted to the original HiT-MDP with the same optimal action values, so abstract-space training sacrifices no optimality.","If Theorem 16 holds, running VMOC on the abstract MDP maximizes a lower bound on the original ELBO, making the abstract variational objective a principled surrogate for the original control problem.","Because options are embeddings rather than triples of initiation set, intra-option policy, and termination function, VMOC avoids the usual computational cost of option-critic architectures and scales to large continuous state-action spaces.","The cold-start SFT phase gives the latent option space a meaningful initialization from human reasoning demonstrations, so an LLM can start with a library of reasoning primitives and refine them through reinforcement learning rather than learning from scratch."],"supporting_citations":[{"why":"Supplies the continuous MDP homomorphism theory that the paper extends to option-based HiT-MDPs, including optimal value equivalence and policy lifting.","marker":"[48]"},{"why":"Introduces HiT-MDP as a Markovian reformulation of the SMDP option framework, so options can be represented as embeddings, and proves homomorphic equivalence to the SMDP form.","marker":"[44]"},{"why":"Defines the option framework and the semi-Markov decision process that VMOC replaces with a latent-variable HiT-MDP.","marker":"[63]"},{"why":"Provides the soft actor-critic off-policy maximum-entropy framework and the soft policy iteration theorem that VMOC adapts to option policies.","marker":"[26]"},{"why":"Provides the control-as-inference formulation and the risk-seeking correction via structured variational inference that underlies the ELBO derivation.","marker":"[39]"},{"why":"Supplies the variational inference machinery, including the evidence lower bound and EM interpretation, used for both VMOC and cold-start training.","marker":"[36]"},{"why":"Establishes the conditional independencies of the HiT-MDP that the induction proof of Theorem 11 relies on when factoring policies over state, action, and option.","marker":"[45]"}],"fun_headline_variants":["Abstraction preserves optimality in option-based RL","Latent options keep optimal policy intact","Variational homomorphisms make abstraction safe","Optimal policies survive latent abstraction","Abstract MDPs: Learn fast, stay optimal"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof's load-bearing premise is that the abstraction map chosen at the current state can also relabel actions at the next state, even though the definition only supplies a separate map per state with no requirement that the maps agree.","fun_headline_variants_meta":{"raw":{"variants":["Abstraction preserves optimality in option-based RL","Latent options keep optimal policy intact","Variational homomorphisms make abstraction safe","Optimal policies survive latent abstraction","Abstract MDPs: Learn fast, stay optimal"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1250,"prompt_tokens":992,"completion_tokens":258,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":191}},"tokens_in":608,"tokens_out":258,"duration_ms":2940,"temperature":1.0,"reasoning_tokens":191,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:09:31.916499+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a two-state HiT-MDP with overlapping action-option sets and choose surjective abstraction maps $g_{e_1}$ and $g_{e_2}$ that disagree on a shared action. If $Q^*(e_1,a,o) \\neq \\bar{Q}^*(f(e_1), g_{e_1}(a,o))$ while Definition 8 is satisfied exactly as written, the claimed equality is false.","supporting_citations":[{"cited_title":"Policy gradient methods in the presence of symmetries and state abstractions.Journal of Machine Learning Research, 25(71):1–57, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the continuous MDP homomorphism theory that the paper extends to option-based HiT-MDPs, including optimal value equivalence and policy lifting."},{"cited_title":"Hit-mdp: learning the smdp option framework on mdps with hidden temporal embeddings","cited_arxiv_id":null,"evidence_quote":"Introduces HiT-MDP as a Markovian reformulation of the SMDP option framework, so options can be represented as embeddings, and proves homomorphic equivalence to the SMDP form."},{"cited_title":"S., Precup, D., and Singh, S","cited_arxiv_id":null,"evidence_quote":"Defines the option framework and the semi-Markov decision process that VMOC replaces with a latent-variable HiT-MDP."},{"cited_title":"and Friedman, N.Probabilistic graphical models: principles and techniques","cited_arxiv_id":null,"evidence_quote":"Supplies the variational inference machinery, including the evidence lower bound and EM interpretation, used for both VMOC and cold-start training."},{"cited_title":"Hit-mdp: learning the smdp option framework on mdps with hidden temporal embeddings","cited_arxiv_id":null,"evidence_quote":"Establishes the conditional independencies of the HiT-MDP that the induction proof of Theorem 11 relies on when factoring policies over state, action, and option."}],"review_version":1}