{"id":"dac16439-8527-4426-969e-16b8d453b018","arxiv_id":"2412.03111","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A metacognitive reinforcement-learning model fits human strategy discovery better than rational strategy selection or mental habits, yet under-predicts how quickly people discover the new planning strategy.","lead":"The authors tested whether people invent new planning strategies through metacognitive reinforcement learning, using an online maze-planning experiment and five computational models. They found that a metacognitive learning model explains more participants' behavior than alternatives, but it still learns the new strategy more slowly than people do.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed discovery mechanism is not established because the seven task-specific features plus hyperparameters tuned to the optimal click sequence encode the target strategy before learning; the model comparison may therefore favor MCRL for representational, not mechanistic, reasons.","rationale":"The empirical phenomenon is well supported: the proportion of adaptive strategies rises significantly, scores improve, and the paper discloses the main negative evidence, namely that fitted models discover more slowly than humans and that some participants show abrupt 'eureka' improvements. The MCRL family has independent precedent in earlier work, and the mental habit and RSSL alternatives are reasonable. The load-bearing weakness is that the target strategy is represented explicitly: the seven added features are task-level preference and avoidance features, and the capability simulation selects hyperparameters by optimizing for the optimal click sequence. Consequently, the discovery simulation demonstrates expressiveness under favorable tuning, not emergence from experience. The BMS comparison also operates inside this same hand-crafted feature space, so it is partly a test of feature engineering plus weight updating rather than a clean test of metacognitive reinforcement learning. This does not invalidate the within-task empirical comparison, but it makes the central mechanistic claim conditional on a pre-specified feature set and an out-of-sample hyperparameter selection. Because the reader already assigned CONDITIONAL with moderate confidence and flagged the feature-set span as the weakest assumption, my stress-test reinforces that conditionality without moving the verdict. The decisive check is to repeat the analysis with the original 56-feature set and a held-out hyperparameter fit; if the advantage over mental habit disappears, the central claim reduces to representational engineering rather than metacognitive learning.","tokens_in":8304,"tokens_out":5888,"duration_ms":65440,"concrete_test":"Re-run the capability and model-comparison analyses using the original 56-feature representation from Jain et al. (2022) without the seven task-specific features added here. First check whether the resource-rational strategy is representable in that reduced space; if it is not, the new features are doing the discovery work. If it is representable, fit the hybrid and model-free Reinforce models with hyperparameters selected on a training half of participants and evaluate BMS on the held-out half. If MCRL no longer matches or outperforms the mental habit model, the reported advantage depends on task-specific feature engineering and should not be read as evidence for a general mechanism. A stronger variant is to pre-register both the feature set and the hyperparameter search before seeing any data from this task.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that MCRL discovers a genuinely new planning strategy. The support rests on two steps: capability simulations showing Reinforce can discover the adaptive strategy 'from scratch' (Figure 3), and BMS showing MCRL explains more participants than alternatives (Table 4). Both are weakened by the same design choice. The 63-feature space includes seven features added for this task, described as 'essential to capture preferences and avoidance related to specific node levels,' and the capability simulations use 'hyperparameters that were found by optimizing for the optimal click sequence.' The target strategy is therefore explicitly engineered into the representational envelope, and the simulation is tuned to produce it. Showing that weights converge to that strategy under these conditions is evidence of expressiveness and favorable tuning, not evidence that a new strategy emerges from experience. Furthermore, the mental habit model shares the same feature set and differs from model-free Reinforce mainly in whether weights update, so the BMS advantage may reflect the added flexibility of weight updating within a hand-crafted space rather than metacognitive value learning as a cognitive mechanism. The authors honestly report that fitted models discover more slowly than humans, but that disclosure does not resolve the deeper issue: within an engineered feature space, a slower discovery rate may reflect the shape of the engineered space rather than the learning mechanism. For the central claim to hold, the feature set must be independently justified and the discovery result must survive out-of-sample conditions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes that new planning strategies are discovered via metacognitive reinforcement learning (MCRL). It introduces a Mouselab-MDP experiment in which participants must learn a task-specific resource-rational (RR) strategy, shows that the proportion of adaptive strategies increases significantly over 120 trials, and then fits two MCRL variants (hybrid and model-free Reinforce) against alternative accounts (RSSL, mental habit formation, and a non-learning model) using family-level Bayesian model selection. The BMS results favor learning models over non-learning, and Reinforce over RSSL and mental habit, with the hybrid variant best among the Reinforce models. The paper also demonstrates, via simulations, that the Reinforce models can in principle discover the adaptive strategy, but acknowledges that when fitted to human data the models discover at a slower rate than participants.","tokens_in":8543,"tokens_out":5928,"duration_ms":53095,"significance":"If the central claim holds, the paper would provide a mechanistic account of how new planning heuristics enter the mental toolbox, extending prior work on strategy selection to strategy discovery. The experimental design is a clear contribution: it creates a novel, task-specific RR strategy that participants are unlikely to know in advance, and the empirical trend (rising proportion of adaptive strategies) is supported by both a logistic regression and a non-parametric Mann-Kendall test. The paper also reports model comparison using random-effects BMS and gives credit for the honest discussion of the models' slower discovery rate and the possible role of 'Eureka' moments. However, the significance is tempered by two load-bearing concerns: the capability simulations use hyperparameters optimized to produce the optimal click sequence, and the seven task-specific features added to the strategy representation are not described in the manuscript, so the representational envelope may be engineered to contain the target strategy. These issues limit the strength of the evidence for MCRL as the actual mechanism of human strategy discovery.","major_comments":[{"comment":"The claim that the Reinforce models 'are capable of discovering the adaptive strategy from scratch' is not supported by the simulations as reported, because the hyperparameters were 'found by optimizing for the optimal click sequence.' This is a circular step: the simulation is tuned to produce the target sequence, so convergence to that sequence demonstrates representational capacity and favorable tuning, not discovery from experience. To support the discovery claim, the authors should show that the model discovers the strategy for a broad, principled range of hyperparameters (e.g., random search, default values, or cross-validated fits) and that the optimized hyperparameters are not special.","section":"Capability to represent and to discover the novel planning strategy (Figure 3)"},{"comment":"The manuscript does not describe the seven new features added for this task; it merely states that they are 'essential to capture preferences and avoidance related to specific node levels' and provides an OSF link. Because the target RR strategy specifically involves examining immediate nodes and one outer node on the positive branch, these features could, in the worst case, directly encode the target strategy. This concern is load-bearing for both the capability demonstration and the model comparison: if the feature space is engineered to contain the target, then the BMS advantage of Reinforce over the mental habit model (which shares the same features) may reflect the flexibility of weight updating in a hand-crafted space rather than metacognitive value learning. The authors should list the seven features in the paper, explain what they capture, and ideally show that the target strategy is not representable by a small subset of them.","section":"Strategy representation (paragraph on the 63-feature set)"},{"comment":"The mental habit model is implemented as a Reinforce model that uses the same features as the model-free variant but does not update the feature weights. Thus the Level-2 BMS contrast (Reinforce vs. mental habit) reduces to updating versus not updating the same feature weights. This comparison does not isolate metacognitive value learning as the mechanism; any time-varying weight model could fit better because it can track changes in behavior. To strengthen the mechanistic interpretation, the authors should add a control model that updates weights without value-based reinforcement (e.g., random drift of weights) or otherwise demonstrate that the value-based update rule is essential for the empirical advantage.","section":"Model fitting and model selection (Tables 3 and 4)"}],"minor_comments":[{"comment":"Equation (2) sums over 56 features, whereas the text states that the feature set has 63 features; please reconcile this discrepancy.","section":"Equation (2)"},{"comment":"The text reports that the proportion of adaptive strategies increased from 0.79% in the first trial to 28.57% in the last trial, but the reported 95% confidence interval for the first trial is [0; 2.22%], which includes zero; the authors should acknowledge this when interpreting the starting point.","section":"Empirical results (first-trial proportion)"},{"comment":"The classification of a click sequence as 'adaptive' is described only in prose; please provide a precise, algorithmic definition (e.g., exact conditions on click orders and counts) so that the classification is reproducible.","section":"Materials (adaptive strategy definition)"},{"comment":"Reference 'He, Jain, & Lieder, 2021b' has a typo in the title: 'planning strategies the to structure of the environment' should be 'planning strategies to the structure of the environment'.","section":"References"},{"comment":"The captions of Figure 2 and Figure 3 are brief; please state the meaning of the shaded areas, the number of simulation runs, and the specific hyperparameter values for each curve in the captions.","section":"Figures 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The paper repeatedly points to an OSF repository (https://osf.io/g3tzp) for essential details: the seven added features, the hyperparameters used in simulations, and the regression analyses for the slower discovery rate. The editors should ensure that this repository is properly anonymized for review and contains all referenced materials. Also, the paper would be a better fit for a journal that values cognitive modeling and resource-rational analysis; if the target venue is an AI journal, the authors may need to strengthen the AI-facing contribution (e.g., broader applicability of the discovery mechanism)."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The empirical core is genuinely new and mostly solid: a novel Mouselab-MDP task where the resource-rational strategy is unlikely to be in participants' toolbox, 349 participants, and a clear rising proportion of adaptive strategies (0.8% to 28.6%, with significant logistic and Mann-Kendall trends). That is a useful result for anyone studying strategy discovery. The paper also deserves credit for honest reporting: the fitted models discover more slowly than humans, and the authors explicitly flag that the feature set may miss intermediate strategies.\n\nWhere I agree with the stress-test: the capability simulation (Figure 3) is not strong evidence for 'from scratch' discovery. The hyperparameters were optimized to produce the optimal click sequence, and seven new features were added because they are 'essential to capture preferences and avoidance related to specific node levels' — in other words, the target strategy is engineered into the representational envelope. Showing that Reinforce converges under those conditions demonstrates expressiveness and favorable tuning, not emergent discovery. The BMS comparison is also weaker than it looks. The mental habit model is a Reinforce with frozen weights; the comparison partly tests whether updating helps within a hand-crafted feature space, not whether metacognitive value learning is the mechanism. The post-hoc explanation of habitual participants (the 'eureka' subgroup) is speculative, though honestly labeled.\n\nStill, the paper is not broken. The intermediate conclusions hold: experience-driven discovery occurs, and among the tested mechanisms, weight-updating Reinforce explains more participants than the alternatives. The abstract's stronger claim — that MCRL is the mechanism — is not established, but the authors partially walk it back in the discussion.\n\nThe paper deserves a serious referee. The right path is an out-of-sample or pre-registered test with independent feature justification and stronger alternative models. I would cite the empirical phenomenon and the task design.","headline":"A solid empirical demonstration of experience-driven strategy discovery, but the central mechanistic claim rests on a tuned simulation and a hand-crafted feature space, so it is not yet established.","tokens_in":9111,"tokens_out":1821,"would_cite":true,"duration_ms":18552,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"People discover new planning strategies by learning which mental operations pay off, not just by choosing among familiar ones.","keywords":["metacognitive learning","planning","strategy discovery","cognitive modeling","reinforcement learning","Mouselab-MDP","resource rationality"],"falsifier":"Simulate click sequences from a known mental-habit model with an abrupt switch to the adaptive strategy, then run the paper's Bayesian model selection on those simulated data; if the procedure attributes a majority of these simulations to hybrid Reinforce, the reported preference for metacognitive RL over habit learning is not diagnostic.","tokens_in":8065,"feed_emoji":"🧠","tokens_out":6074,"duration_ms":60161,"temperature":0.7,"pith_summary":"This paper asks where planning strategies come from, not just how people choose among strategies they already have. It proposes that new strategies are discovered through metacognitive reinforcement learning: people treat mental operations, such as clicking to reveal a node in a planning tree, as actions and update preferences for those operations based on the rewards they bring. In a custom Mouselab-MDP planning task where the resource-rational strategy was deliberately unfamiliar, the proportion of participants using the adaptive strategy rose from 0.79% in the first trial to 28.57% by trial 120, and scores rose accordingly. When fitted to individual click sequences, metacognitive reinforcement learning models explained more participants than strategy-selection learning, value-free mental habit formation, or no learning, with 61% best explained by the reinforcement mechanism and the hybrid variant preferred within it. The models nevertheless discovered the strategy more slowly than people did, which the authors attribute to missing \"Eureka\"-style insight learning and possibly to a feature representation that does not cover every intermediate strategy.","feed_headline":"Planning strategies emerge via metacognitive reinforcement learning","feed_subtitle":"Fitted to 349 people, meta-level trial and error beats strategy-choice and habit models, but humans still learn faster.","key_machinery":"The central object is the metacognitive reinforcement learning model, specifically the Reinforce algorithm applied to a meta-level Markov decision process whose states are mental belief states and whose actions are mental computations such as revealing a node. Planning strategies are represented by 63 features of belief states and cognitive operations, with weights approximating meta-level Q-values; the next planning operation is chosen via a softmax over those Q-values, and after each trial the weights are updated by the Reinforce gradient-ascent rule using the return of that trial's planning episode. The load-bearing mechanism is this gradient ascent in strategy space: starting from an arbitrary initial weighting, repeated updates should move the model toward the resource-rational strategy, and the model comparison then tests whether human click sequences follow the same update rule.","core_discovery":"The paper claims that experience-driven strategy discovery in planning is best explained by metacognitive reinforcement learning (MCRL), in which the brain learns to approximate an optimal meta-level policy by updating feature weights attached to cognitive operations. To test this, the authors designed a Mouselab-MDP environment whose resource-rational strategy—check immediate nodes until the branch with a positive start is identified, then inspect exactly one outer node on that branch—was unlikely to be in participants' pre-existing mental toolboxes. Across 120 trials, the proportion of participants using the adaptive strategy increased significantly, and Bayesian model selection over click sequences favored the hybrid Reinforce model for about 62% of the participants best described by a Reinforce variant, with rational strategy selection learning explaining under 1% overall. The authors conclude that metacognitive reinforcement learning is a key mechanism for discovering new planning strategies, while acknowledging that their models have a slower discovery rate than humans and that individual differences exist, with about 39% of participants better described by a value-free mental habit model.","pith_inferences":["Editorial inference: if metacognitive reinforcement learning generalizes beyond this task, training environments could be designed so that the cost structure of mental operations is discoverable, potentially accelerating planning-skill acquisition in education and AI.","Editorial inference: the slower-model-faster-human gap suggests a low-probability \"insight\" sampler that reinitializes feature weights when cumulative reward is poor would improve the model; this is a testable extension the paper does not implement.","Editorial inference: the 21 habitual participants who examined all nodes early and then snapped to the optimal strategy could be reinterpreted as performing deliberate exploration first; modeling this as an explore-then-exploit meta-policy is a natural extension that current gradual-learning models cannot capture."],"forward_implications":["Strategy discovery is experience-driven: the adaptive strategy was almost never present on trial 1 (0.79%) but was used by 28.57% of participants by trial 120.","Metacognitive reinforcement learning is a better explanation of strategy discovery than choosing among pre-existing strategies: rational strategy selection learning explained under 1% of participants, while the Reinforce mechanism accounted for about 61%.","Strategy acquisition is not a single mechanism for everyone: about 39% of participants were better explained by value-free mental habit formation than by learning from reward feedback.","Current MCRL models under-predict discovery speed, so a full account of human strategy discovery must include something like insight-driven \"Eureka\" jumps or active learning.","Because the models learn from feedback on mental computations, interventions aimed at improving planning should target the value of thinking itself, not only the value of external actions."],"supporting_citations":[{"why":"Supplies the metacognitive reinforcement learning framework that the paper applies to strategy discovery.","marker":"Callaway et al., 2018"},{"why":"Introduced the Mouselab-MDP paradigm whose click sequences are the paper's behavioral measure.","marker":"Callaway et al., 2017"},{"why":"Provided the base feature representation for planning strategies and the set of 79 strategies used by the RSSL comparison.","marker":"Jain et al. (2022)"},{"why":"The Reinforce gradient-following algorithm behind the weight update rule of the MCRL models.","marker":"Williams (1992)"},{"why":"Defined the rational strategy selection learning model that serves as the main alternative mechanism.","marker":"Lieder & Griffiths (2017)"},{"why":"Provides evidence for value-free habit formation on which the mental habit model is built.","marker":"Miller et al. (2019)"},{"why":"Used dynamic programming to confirm that the click pattern taught in the task is the resource-rational strategy.","marker":"Callaway et al., 2022"},{"why":"Provides the Bayesian model selection method used to compare model families across participants.","marker":"Rigoux et al., 2014"}],"fun_headline_variants":["Metacognitive learning reveals new planning strategies","Meta-level trial-and-error discovers planning strategies","Humans outpace model in planning strategy discovery","How planning strategies are born: metacognitive reinforcement","Experience drives strategy discovery via meta-reinforcement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the 63-feature strategy space (with seven features added for this task) can represent every intermediate strategy participants actually tried; if a real intermediate strategy falls outside this representation, the model can only discover strategies within its own vocabulary and the comparison loses force.","fun_headline_variants_meta":{"raw":{"variants":["Metacognitive learning reveals new planning strategies","Meta-level trial-and-error discovers planning strategies","Humans outpace model in planning strategy discovery","How planning strategies are born: metacognitive reinforcement","Experience drives strategy discovery via meta-reinforcement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001061,"raw_usage":{"total_tokens":4414,"prompt_tokens":870,"completion_tokens":3544,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":3475}},"tokens_in":486,"tokens_out":3544,"duration_ms":25853,"temperature":1.0,"reasoning_tokens":3475,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:44:43.486506+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate click sequences from a known mental-habit model with an abrupt switch to the adaptive strategy, then run the paper's Bayesian model selection on those simulated data; if the procedure attributes a majority of these simulations to hybrid Reinforce, the reported preference for metacognitive RL over habit learning is not diagnostic.","supporting_citations":[{"cited_title":", Gul, S","cited_arxiv_id":null,"evidence_quote":"Supplies the metacognitive reinforcement learning framework that the paper applies to strategy discovery."},{"cited_title":", Lieder, F","cited_arxiv_id":null,"evidence_quote":"Introduced the Mouselab-MDP paradigm whose click sequences are the paper's behavioral measure."},{"cited_title":", Callaway, F","cited_arxiv_id":null,"evidence_quote":"Provided the base feature representation for planning strategies and the set of 79 strategies used by the RSSL comparison."},{"cited_title":"APACrefauthors \\ 1992","cited_arxiv_id":null,"evidence_quote":"The Reinforce gradient-following algorithm behind the weight update rule of the MCRL models."},{"cited_title":"\\ Griffiths, T L","cited_arxiv_id":null,"evidence_quote":"Defined the rational strategy selection learning model that serves as the main alternative mechanism."},{"cited_title":", Shenhav, A","cited_arxiv_id":null,"evidence_quote":"Provides evidence for value-free habit formation on which the mental habit model is built."},{"cited_title":", van Opheusden, B","cited_arxiv_id":null,"evidence_quote":"Used dynamic programming to confirm that the click pattern taught in the task is the resource-rational strategy."},{"cited_title":", Stephan, K E","cited_arxiv_id":null,"evidence_quote":"Provides the Bayesian model selection method used to compare model families across participants."}],"review_version":1}