{"id":"c8133f13-ddda-4adc-9101-a80b71f9c48e","arxiv_id":"2411.11511","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A variational Gaussian mixture with an online component-pruning and matching mechanism is combined with a Dirichlet-categorical transition model and belief-based Q-learning to solve small mazes from continuous observations.","lead":"This paper introduces a temporal Gaussian mixture model that learns discrete states from continuous observations while an agent explores a maze, and couples it to Q-learning for navigation. The authors report that the model discovers state counts and transitions in small mazes, and that it is competitive with DQN and A2C on several of them.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Component identity drift undermines the structure-learning claim; the paper's own Figure 12 shows components fusing and being unlearned as state count grows.","rationale":"The reader's weakest assumption correctly identifies the load-bearing condition: transition and forgetting machinery presuppose that Gaussian components stay matched to the same physical maze cell over time. I agree with this assessment. The paper is honest enough to include the evidence against it in Figure 12 and Section 7, which strengthens the concern rather than weakening it. If component identity drifts, the 'states' over which B and Q-values are learned are not the environment's states, so the abstract's claim that the model discovered the number of states and transition probabilities is not established. I considered whether the unspecified online component-addition procedure is an even more fundamental gap, since the paper never gives equations or pseudocode for adding components, but that is a reproducibility and completeness issue that could be resolved by releasing code. The identity-drift concern is a demonstrated empirical failure of the central mechanism, so it is the more load-bearing objection. The standard variational and Dirichlet-categorical machinery is competently derived, and the small-maze results provide partial support, so a conditional verdict remains appropriate rather than rejection.","tokens_in":36852,"tokens_out":8611,"duration_ms":100905,"concrete_test":"Re-run the maze experiments while logging, every 100 training steps, the Gaussian component whose mean is nearest to each maze cell, together with the component persistence count of Section 4.2. For each maze in Figure 12, compute the fraction of the final 1000 training steps for which every cell on the successful path is matched to the same persistent component, and check whether any component mean crosses a cell boundary. If any cell on the solution path has a matching fraction below 0.95, or if component means cross into neighboring cells, then the learned transition matrices are not a model over the maze's true states, and the reported navigation success cannot support the structure-learning claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the model discovers the number of states and the transition probabilities between them is only meaningful if Gaussian components remain identifiable as the same physical state over time. The fixed-component mechanism of Section 4.2 matches components across 100-step windows by KL divergence below 0.5, and forgetting in Section 4.3 is permitted only for observations associated with fixed components. Once a component drifts to a neighboring cell or two components fuse, the Dirichlet-categorical transition counts in Equations (55)-(56) accumulate over clusters that are not states, so the learned B matrices are not transition probabilities of the environment. The paper's own experiments demonstrate exactly this failure: Figure 12 reports orange cells whose components become unstable and take over neighboring cells, and red cells that are unlearned by the end of simulation; Section 7 concedes that 'several components fuse together into a single component.' Maze (f) is unsolved precisely because the large number of states renders the Gaussian components unstable. Thus the abstract's claim that the model discovered the number of states and the transition probabilities is only supported for the smallest mazes, and even there the successes rest on a stability condition that the paper shows does not hold generally.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a temporal Gaussian mixture model (TGM) for model-based reinforcement learning in continuous-observation environments. The perception model is a variational Gaussian mixture (VGM) initialized by mean-shift clustering, with an empirical-prior mechanism that lets the model forget part of the data while integrating its information into the prior. The transition model is a Dirichlet-categorical model that accumulates transition counts between inferred Gaussian components under each action. A belief-scaled Q-learning variant is introduced for planning from posterior beliefs over latent states. The experimental section evaluates the model on six small maze environments, comparing against DQN and A2C, and reports that the TGM learns the maze structure and solves a subset of the mazes. Extensive variational derivations are provided in Appendices B--F.","tokens_in":37135,"tokens_out":4470,"duration_ms":49291,"significance":"If the central claims are fully supported, the paper would make a useful contribution: it combines a tractable variational Gaussian mixture with a forgetting mechanism and a belief-based Q-learning update, providing an interpretable model-based alternative to model-free deep RL on small continuous-state tasks. The variational derivations in the appendices are detailed and mostly standard, which is a strength: the update equations for the perception and transition models are given explicitly and are checkable. However, the paper's central empirical claim—that the model 'discovered the number of states and the transition probabilities'—is currently supported only by qualitative inspection of small mazes, and the paper's own results show that component identity drift causes failures as the number of states grows. The incremental contribution over existing variational GMM and active-inference structure-learning work is real but modest, and the lack of quantitative structure-learning metrics, convergence guarantees for the belief Q-learning update, and code release limits the paper's impact.","major_comments":[{"comment":"The component-persistence and fixing mechanism is described only procedurally, with ad-hoc thresholds θkl=0.5 and θcounts=4. The paper does not specify how component identities are tracked when the number of components K grows or shrinks, nor how the KL matching is performed when components split or merge. Since forgetting (Section 4.3) and transition-count accumulation (Eqs. (55)--(56)) are restricted to fixed components, any matching failure propagates directly into the learned transition tensor B. The paper's own Figure 12 and Section 7 concede that components become unstable, take over neighboring cells, and fuse together. Therefore the abstract's claim that the model discovered the number of states and transition probabilities is not supported for general mazes. Please provide a quantitative, ground-truth-based evaluation of structure discovery (e.g., component-to-cell confusion matrices, component count over time, number of identity switches) for every maze, and constrain the abstract's claims accordingly.","section":"Section 4.2, Eq. (57)"},{"comment":"In the transition model, the variational factors Q(Z0) and Q(Z1) are initialized from the perception model and are explicitly kept fixed; no update equations for Z0 and Z1 are derived for the transition model. This means the transition model never uses action or transition information to refine the state posterior, so the model is not a single jointly inferred temporal model but two independently fitted components. The empirical statement in the text that separate learning 'works better' is not quantified. Please state this as an explicit approximation, justify it with an ablation, or derive the missing update equations.","section":"Section 3.1, Eqs. (52) and surrounding text"},{"comment":"The belief-scaled Q-learning update is introduced without convergence analysis. Standard Q-learning convergence requires repeated updates of each state-action pair under a Robbins-Monro step-size schedule, but here the update is scaled by the posterior Q(zt), the transition kernel P(zt+1|zt,at) is itself estimated and changing over time, and the state set K grows and shrinks. The paper does not provide a fixed-point or contraction argument, and the experiments do not isolate the contribution of the belief weighting. Please add a convergence or consistency analysis, or at least an ablation comparing Eq. (71) to standard Q-learning with a hard assignment to the maximum-responsibility state.","section":"Section 5.2, Eq. (71)"},{"comment":"The evaluation of structure learning relies on manual inspection of learned components ('By manual inspection of the learned components and transition matrices...'), and the unsolved mazes are attributed to exploration and component instability without quantitative support. The number of independent runs, hyperparameter settings, and the exact state representation used for DQN and A2C are not reported. There is also no model-based baseline. Please report quantitative structure-learning metrics (e.g., adjusted Rand index against the true cell partition, learned state count over time), standard errors over seeds, and the full hyperparameter configurations for all algorithms.","section":"Section 6, Figure 12 and Figure 13"}],"minor_comments":[{"comment":"The KL divergence between two Gaussian distributions is asymmetric, but the direction is not specified. Please state explicitly whether the divergence is DKL[N(µ1,Λ1)||N(µ2,Λ2)] or the reverse, and whether the comparison is symmetric in practice.","section":"Section 4.2, Eq. (57)"},{"comment":"The text says that when a component's responsibilities become zero, 'the posterior parameters of this component revert to their prior counterparts,' but this reversion is not reflected in the update equations. Please specify the condition and the exact parameters that are reverted.","section":"Section 2.2.4, after Eq. (41)"},{"comment":"The index order in the transition-count updates (k for z0 and j for z1) appears inconsistent with the categorical likelihood in Eq. (77), where the parameters B[a] are indexed with zτ first and zτ+1 second. Please check and unify the index conventions.","section":"Section 3.1, Eqs. (55)--(56) and Eq. (77)"},{"comment":"The hyperparameters dk, βk, and vk are set to values proportional to K, but K changes over time as components are added or removed. The paper does not discuss how the priors should be rescaled or whether this dependence is intentional; a brief comment would help.","section":"Section 2.2.1"},{"comment":"The caption states that state seven is above state six but not displayed because the agent did not visit it in the last 100 steps. This is confusing: if the state is never visited, it is unclear how it was learned; please clarify the visitation condition and its effect on the displayed transition matrices.","section":"Figure 7 caption"},{"comment":"In Figure 13(b) the text says all three agents performed the same, and only DQN is visible. Please state explicitly whether the TGM and A2C curves are exactly zero or whether they were omitted for legibility, and clarify the episode-reward scale used for the comparison.","section":"Section 6, Figure 13"},{"comment":"The conditional expectation Eπ[rt+1 + γGt+1 | st+1] omits the conditioning on st and at. As written it is a notational shortcut that may mislead readers about the derivation; please write the full conditioning.","section":"Appendix F, Eq. (179)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is borderline for a full journal paper. The variational derivations are thorough, but the central empirical claims are not quantitatively substantiated: structure learning is evaluated manually, component identity drift is acknowledged and visible in the paper's own figures, and the belief Q-learning update has no convergence analysis. I would encourage the authors to add a rigorous quantitative evaluation, an ablation isolating the effects of the forgetting and belief mechanisms, and a clearer statement of the scope of the claims. Also, no code is released, which makes reproducibility difficult for a paper with this many algorithmic components. The related-work discussion is heavily weighted toward active inference; positioning against nonparametric Bayesian RL methods (e.g., infinite HMMs, Bayes-adaptive RL) would help readers assess novelty."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on the TGM paper. The genuinely new bit is the full stack: a variational Gaussian mixture perception model with online component addition, a Dirichlet-categorical transition model, and belief-scaled Q-learning. The variational derivations in the appendices are extensive and largely standard, and the authors are honest about where the method fails. That honesty is real credit.\n\nThe main soft spot is the gap between the abstract and what the experiments actually support. The structure-learning claim depends on Gaussian components staying matched to physical maze cells over time. The paper's own mechanism matches components every 100 steps by KL divergence, fixes them after four consecutive matches, and then forgets data only around fixed components. But Figure 12 shows orange cells whose components drift to neighboring cells and red cells that get unlearned, and Section 7 concedes that components fuse. Once component identity drifts, the transition counts in Equations (55)-(56) accumulate over clusters that are not states, so the learned B matrices are not transition probabilities of the environment. Maze (f) is unsolved for exactly this reason. So the abstract's statement that the model discovered the number of states and transition probabilities is only supported for the smallest mazes, and even there the success depends on a stability condition that the paper shows does not hold generally.\n\nA second soft spot is that the central online component-addition mechanism is described only in prose in Section 4.2. There is no pseudocode or formal update for when and how new components are inserted. That hurts reproducibility. The experiments also lack detail: six small mazes, manual inspection to label cells as learned or unlearned, no number of seeds reported, and no hyperparameter sensitivity for the new thresholds. The belief-scaled Q-learning is heuristic, with no convergence analysis, though the paper does not overclaim this.\n\nAll that said, the core variational machinery is sound, the paper is clearly written, and the limitations section is unusually candid. This is a legitimate methods contribution that would benefit from a revision that formalizes the component-addition step and tests on a broader set of environments with more quantitative evaluation. It is not a desk-reject; a serious referee can help the authors turn this into a solid publication.","headline":"A transparent but partial structure-learning result: the method works on small mazes, and the paper's own figures show the component-drift problem that limits the central claim.","tokens_in":37608,"tokens_out":1692,"would_cite":false,"duration_ms":18748,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A temporal Gaussian mixture with a Dirichlet-categorical transition model and belief-scaled Q-learning can discover the number of maze states, learn transition probabilities, and navigate from start to exit.","keywords":["structure learning","temporal Gaussian mixture","variational Gaussian mixture","model-based reinforcement learning","belief-based Q-learning","Dirichlet-categorical transition model","continuous observation maze navigation"],"falsifier":"Record the maze cell occupied by the agent and the Gaussian component with highest responsibility at every 100-step checkpoint. If the same physical cell is ever assigned to two different components that satisfy the $D_{\\mathrm{KL}} < 0.5$ matching criterion, or if a fixed component's mean shifts from one cell to a neighbor while staying fixed, the identity assumption fails and the learned transition probabilities and Q-values describe a different environment; the paper's Figure 12 already shows unlearned components in mazes 12(b) and 12(f).","tokens_in":36653,"feed_emoji":"🧀","tokens_out":7355,"duration_ms":68638,"temperature":0.7,"pith_summary":"The paper proposes a model-based reinforcement learner that builds its own map of a continuous environment. The core claim is that a temporal Gaussian mixture—a variational Gaussian mixture for perception plus a Dirichlet-categorical model for transitions—can discover how many states a maze has, learn the transition probabilities between those states, and then plan with a Q-learning rule that works on belief states rather than observed states. The authors demonstrate the claim in several small continuous mazes: the learned components line up with maze cells, the learned transition tables match the environment's dynamics, and the agent navigates from start to the exit. The reason to care is that the number of latent states is not given in advance, so the same machinery could apply where an expert model is unavailable.","feed_headline":"Model learns maze structure with no known state count","feed_subtitle":"A temporal Gaussian mixture discovers states and transitions; belief-scaled Q-learning gets the agent to the exit.","key_machinery":"The carrying object is the temporal Gaussian mixture (TGM), which splits into two conjugate-friendly models. The perception model is a variational Gaussian mixture (VGM): each latent state is a multivariate Gaussian component, mean-shift clustering initializes the components, and variational updates compute responsibilities $\\hat r_{nk}$, with unused components pruned and new components added as data accumulate. The transition model is a Dirichlet-categorical network whose concentration tensor $\\hat b[a]_{kj}$ counts, probabilistically, transitions from component $k$ to component $j$ under action $a$; conjugacy makes the updates simple accumulations $\\hat b[a]_{kj} = \\bar b[a]_{kj} + \\sum_{n\\in M''} [a=a^0_n]\\hat r^0_{nk}\\hat r^1_{nj}$. The forgetting mechanism is what keeps the method online: observations associated with fixed components are folded into an empirical prior (same functional form as the original prior) and then discarded, while flexible components keep their data. Structure learning is driven by the fixed/flexible distinction: every 100 iterations the model matches components by KL divergence below 0.5, and a component becomes fixed only after persisting four consecutive checks. The Q-learning variant then uses the learned $P(z_{t+1}|z_t,a_t)$ and the posterior belief $Q(z_t)$ to update values: $q(a_t,z_t)\\leftarrow q(a_t,z_t)+\\alpha Q(z_t)[r_{t+1}+\\gamma\\sum_{z_{t+1}}P(z_{t+1}|z_t,a_t)\\max_{a_{t+1}}q(a_{t+1},z_{t+1})-q(a_t,z_t)]$.","core_discovery":"On its own terms, the paper's discovery is that structure learning and planning can be combined without a fixed state count. The perception model is a variational Gaussian mixture initialized by mean-shift clustering: it prunes Gaussian components that no data support and adds new components whenever the agent encounters a new cluster, so the number of components (states) is learned rather than specified. A separate categorical-Dirichlet transition model records, for each action, the probabilistic counts of moving from one component to the next, and conjugate updates give the transition probabilities $P(z_{t+1}|z_t,a_t)$. To keep memory bounded, the model folds forgettable observations into empirical priors, but only for components that are deemed fixed by repeated KL-based matching across 100-step checks. Finally, decision making uses a variant of Q-learning in which the temporal-difference term is scaled by the posterior belief $Q(z_t)$ over states; the paper shows this reduces to standard Q-learning when the belief is a point mass. Empirically, the agent solved four of six mazes, and the authors report that it discovered the number of states and the transition probabilities.","pith_inferences":["Editorial extension: the KL-threshold matching rule treats component identity as a local, pairwise judgment; an alternative is to track identity by posterior predictive probability over a fixed set of cells, which would let the model detect that a 'fixed' component has drifted to a neighboring cell.","Editorial extension: the forgetting rule keeps only observations attached to fixed components, so memory is automatically concentrated on stable parts of the environment; one could test whether making the KL threshold and persistence count scale with maze size stabilizes larger mazes.","Editorial extension: belief-scaled Q-learning suggests a natural exploration bonus—update Q-values for states in proportion to current belief, then choose actions by uncertainty in the transition model; the paper does not implement this, but its own exploration failure in the corridor maze points to it.","Editorial extension: the paper intentionally trains perception before transitions because joint learning performs worse; a natural test is to re-tune both models together after an initial structure-learning phase, to see whether the VGM's component competition is disrupted by transition information."],"forward_implications":["A model-based RL agent can operate in continuous state spaces without a pre-specified number of states: the perception model adds and prunes Gaussian components as the agent explores.","The learned transition tensor $B[a]_{kj}$ gives an interpretable description of the environment's dynamics, so the resulting behavior is inspectable rather than a black-box policy.","Belief-scaled Q-learning is a viable planning rule under state uncertainty; when the posterior is concentrated on one state, it coincides with standard Q-learning.","On the tested mazes, the method is competitive with DQN and A2C: it outperforms A2C on all solvable mazes, matches or beats DQN on some, and learns faster on one maze.","The structure-learning loop is the bottleneck: with more states, components become unstable and fuse or disappear (orange and red cells in Figure 12), and lack of exploration prevents solving a long-corridor maze."],"supporting_citations":[{"why":"Supplies the variational Gaussian mixture generative model, structured mean-field updates, and the expectation identities used in the perception model's free energy.","marker":"Bishop and Nasrabadi, 2006"},{"why":"Provides mean-shift clustering, used to initialize the Gaussian component parameters without knowing the number of clusters in advance.","marker":"Carreira-Perpinán, 2015"},{"why":"Gives the general variational update equation (23) that the paper applies to derive both the VGM and the Dirichlet-categorical transition updates.","marker":"Winn and Bishop, 2005"},{"why":"Defines standard Q-learning and the return framework that the belief-scaled Q-learning variant extends.","marker":"Sutton and Barto, 2018"},{"why":"Supplies the DQN baseline used in the maze experiments and the idea of learning Q-values as a value function.","marker":"Mnih et al., 2015"},{"why":"Supplies the A2C baseline used in the maze experiments.","marker":"Mnih et al., 2016"},{"why":"Defines the KL divergence used to match Gaussian components across consecutive 100-step checks when deciding whether a component persists and becomes fixed.","marker":"Kullback and Leibler, 1951"}],"fun_headline_variants":["Agent discovers maze states, no preset count needed","Structure-learning RL finds states and routes on the fly","Maze solver learns state count via Gaussian mixture","Belief-scaled Q-learning navigates without known states","Temporal Gaussian mixture teaches agent maze structure"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a Gaussian component which looks the same across consecutive 100-step checks really is the same physical state forever; the forgetting rules and the transition counts all assume this identity is stable, and the paper's own experiments show it fails as the number of maze states grows.","fun_headline_variants_meta":{"raw":{"variants":["Agent discovers maze states, no preset count needed","Structure-learning RL finds states and routes on the fly","Maze solver learns state count via Gaussian mixture","Belief-scaled Q-learning navigates without known states","Temporal Gaussian mixture teaches agent maze structure"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1307,"prompt_tokens":1027,"completion_tokens":280,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":206}},"tokens_in":643,"tokens_out":280,"duration_ms":2859,"temperature":1.0,"reasoning_tokens":206,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:25:51.413408+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record the maze cell occupied by the agent and the Gaussian component with highest responsibility at every 100-step checkpoint. If the same physical cell is ever assigned to two different components that satisfy the $D_{\\mathrm{KL}} < 0.5$ matching criterion, or if a fixed component's mean shifts from one cell to a neighbor while staying fixed, the identity assumption fails and the learned transition probabilities and Q-values describe a different environment; the paper's Figure 12 already shows unlearned components in mazes 12(b) and 12(f).","supporting_citations":[{"cited_title":"Variational message passing","cited_arxiv_id":null,"evidence_quote":"Gives the general variational update equation (23) that the paper applies to derive both the VGM and the Dirichlet-categorical transition updates."}],"review_version":1}