{"id":"f8f3c1d3-c096-424e-8165-8eaa5d54c90f","arxiv_id":"2210.13455","paper_version":6,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"EMCTS modifies MCTS to account for epistemic uncertainty propagation, yielding higher sample efficiency on SUBLEQ code writing and faster solving of Deep Sea variants than standard AlphaZero/MuZero baselines.","lead":"The paper introduces Epistemic Monte Carlo Tree Search (EMCTS) to propagate epistemic uncertainty from learned models through MCTS for improved deep exploration in sparse-reward settings. This could make model-based RL agents more sample-efficient on hard exploration problems such as code synthesis and navigation benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Epistemic uncertainty propagation rule through MCTS lacks derivation or validation, making deep-exploration benefit claims rest on an unverified mechanism","rationale":"The reader's weakest_assumption correctly isolates the missing theoretical link. Because the headline results are attributed to “search for uncertainty estimation,” any gap in how uncertainty is propagated directly undermines the causal claim. The low-confidence UNVERDICTED verdict is therefore appropriate until the derivation and a targeted ablation are supplied.","tokens_in":1693,"tokens_out":310,"duration_ms":11932,"concrete_test":"Extract the exact propagation equations from the methods section; re-derive them from the stated theoretical motivation without additional assumptions; then ablate the propagation step (replace with direct use of leaf uncertainty) and rerun the Deep Sea variants—if performance drops to baseline levels the propagation rule is load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claims (higher sample efficiency on SUBLEQ; solving Deep Sea variants where baselines fail) require that (1) the learned model's epistemic uncertainty estimates are accurate and (2) the specific propagation rule inside EMCTS converts those estimates into reliable deep-exploration bonuses. The abstract asserts a “theoretically motivated” approach but supplies neither the derivation of the propagation operator nor any validation that it preserves the intended uncertainty semantics under MCTS backup. If the rule is heuristic or the estimates degrade under tree search, the reported gains could be artifacts of implementation details rather than the claimed epistemic-search interaction.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes Epistemic Monte Carlo Tree Search (EMCTS) to incorporate epistemic uncertainty from learned models into MCTS backups, enabling deeper exploration in sparse-reward settings. It reports that AlphaZero augmented with EMCTS achieves substantially higher sample efficiency than baseline AZ on the SUBLEQ assembly-code task and solves variants of the Deep Sea hard-exploration benchmark far faster than A/MZ or non-search uncertainty baselines.","tokens_in":1811,"tokens_out":431,"duration_ms":12941,"significance":"If the claimed propagation mechanism is shown to be theoretically sound and empirically reliable, the work would provide a concrete way to turn model uncertainty into search-driven exploration bonuses, addressing a known limitation of the A/MZ family in sparse-reward domains.","major_comments":[{"comment":"§3 (EMCTS algorithm): the epistemic-uncertainty propagation operator through MCTS backups is introduced without a derivation showing that it preserves the intended semantics of the learned model's epistemic estimates; the abstract's claim of a 'theoretically motivated' approach therefore rests on an unverified rule whose correctness is load-bearing for the deep-exploration results.","section":"§3"},{"comment":"§4.2 (Deep Sea experiments): the reported ability of EMCTS to solve variants that baseline A/MZ cannot solve is presented without ablation or diagnostic results confirming that the gains arise from the uncertainty-propagation rule rather than other implementation differences; this leaves the central empirical claim vulnerable to alternative explanations.","section":"§4.2"}],"minor_comments":[{"comment":"Notation for epistemic versus aleatoric uncertainty is introduced in §2 but not consistently distinguished in later equations or pseudocode.","section":"§2"},{"comment":"The SUBLEQ task description would benefit from an explicit statement of the reward sparsity level and state-space size to allow direct comparison with prior work.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments, which highlight important areas for strengthening the manuscript. We address each major point below and will make revisions to improve the theoretical grounding and empirical robustness of the work.","responses":[{"response":"We agree that the manuscript would benefit from an explicit derivation or expanded justification of the propagation operator to better support the 'theoretically motivated' claim. The operator is constructed to propagate epistemic uncertainty estimates through the tree backups in direct analogy to standard value propagation, so that uncertainty at leaf nodes can influence root action selection. However, the current presentation relies on this design motivation without a formal proof that the semantics of the model's epistemic estimates are preserved under the operator. We will revise §3 to include a more detailed derivation or analysis of the operator.","revision_made":"yes","referee_comment":"[§3] §3 (EMCTS algorithm): the epistemic-uncertainty propagation operator through MCTS backups is introduced without a derivation showing that it preserves the intended semantics of the learned model's epistemic estimates; the abstract's claim of a 'theoretically motivated' approach therefore rests on an unverified rule whose correctness is load-bearing for the deep-exploration results."},{"response":"We acknowledge that the Deep Sea results would be more convincing with explicit ablations isolating the contribution of the uncertainty-propagation rule. While the experimental setup was designed to keep all components equivalent except for the propagation mechanism, the manuscript does not include diagnostic comparisons (e.g., an otherwise identical search method with the propagation disabled). We will add such ablation studies to §4.2 in the revision to confirm that the observed gains are attributable to the propagation rule.","revision_made":"yes","referee_comment":"[§4.2] §4.2 (Deep Sea experiments): the reported ability of EMCTS to solve variants that baseline A/MZ cannot solve is presented without ablation or diagnostic results confirming that the gains arise from the uncertainty-propagation rule rather than other implementation differences; this leaves the central empirical claim vulnerable to alternative explanations."}],"tokens_in":1279,"tokens_out":444,"duration_ms":15795,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is that the authors modify MCTS to carry epistemic uncertainty from the learned model through the tree and use that to drive deeper exploration. On SUBLEQ they get noticeably better sample efficiency than plain AlphaZero, and on Deep Sea variants the search version solves problems that the baselines essentially cannot. That is the concrete result a reader should take away first. The new piece is the specific EMCTS backup that treats uncertainty estimates as something to be propagated rather than ignored. Standard A/MZ already use learned models, so the gap they target is real and the idea of letting search itself refine the uncertainty signal is a reasonable next step. The experiments are run on recognized hard-exploration benchmarks, which gives the claims some grounding. The soft spot is exactly the one the stress-test flags. The abstract calls the approach theoretically motivated, yet supplies no derivation of the propagation operator and no check that the operator preserves the meaning of the uncertainty estimates once they are backed up through the tree. If the rule turns out to be a heuristic whose behavior depends on implementation details, the reported gains could be fragile. The paper would be stronger with either a short proof sketch or an ablation that isolates the propagation step from other changes. The citation pattern is standard and does not hide prior work. This is a paper for people who already work on model-based RL and planning under uncertainty. A reader who cares about exploration bonuses or uncertainty-aware search will find the empirical comparison useful even if they end up modifying the propagation rule. It is worth sending to referees because the problem is well-posed, the benchmarks are appropriate, and the results are strong enough to merit scrutiny rather than an immediate desk reject.","headline":"EMCTS adds a propagation step for epistemic uncertainty inside MCTS and reports clear gains on SUBLEQ and Deep Sea, but the rule itself still needs a derivation or validation that the abstract does not supply.","tokens_in":2280,"tokens_out":420,"would_cite":false,"duration_ms":16905,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"Epistemic variance propagation via Taylor/Jacobian in MCTS shares no machinery with RS J-cost forcing or distinction-derived geometry","alignment":"orthogonal","rationale":"Paper derives covariance recursion Σ_{k+1} ≈ Σ(ŝ_k,a_k) + J_f Σ_k J_f^T and UBE-style value uncertainty inside MCTS backups; no J(x)=½(x+x^{-1})−1, no φ-ladder, no 8-tick periodicity, no ratio-symmetric cost, and no parameter-free derivation from a single distinction. Domain (RL planning) lies outside RS structural theorems.","tokens_in":56287,"confidence":"high","tokens_out":146,"duration_ms":6984,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Epistemic Monte Carlo Tree Search propagates model uncertainty through the search tree to improve deep exploration in sparse-reward settings.","keywords":["Epistemic Monte Carlo Tree Search","AlphaZero","exploration","sparse rewards","Monte Carlo Tree Search","reinforcement learning","uncertainty estimation"],"falsifier":"Run EMCTS and a baseline without uncertainty propagation on the same Deep Sea variants; if the baseline solves them at comparable speed or sample count, the claimed benefit of search-based uncertainty handling does not hold.","tokens_in":2603,"feed_emoji":"🔍","tokens_out":557,"duration_ms":11304,"temperature":0.7,"pith_summary":"The paper presents Epistemic MCTS as a way to incorporate epistemic uncertainty from learned models directly into Monte Carlo Tree Search. Standard A/MZ algorithms ignore how this uncertainty should influence search decisions, limiting their ability to explore efficiently when rewards are rare. By using the search process itself to estimate and act on uncertainty, the method yields higher sample efficiency when combined with AlphaZero on a SUBLEQ code-writing task and solves Deep Sea variants that baseline methods cannot handle.","feed_headline":"Epistemic search lifts AlphaZero efficiency on sparse tasks","feed_subtitle":"Propagating model uncertainty inside the tree search solves Deep Sea variants that standard A/MZ cannot handle.","key_machinery":"Epistemic MCTS, a modification to standard MCTS that integrates and propagates epistemic uncertainty estimates produced by the learned model to guide action selection toward regions of high uncertainty.","core_discovery":"Epistemic MCTS accounts for epistemic uncertainty caused by limited training data by propagating these estimates through the Monte Carlo Tree Search, allowing the search to perform deep exploration rather than treating uncertainty as noise or ignoring it entirely.","pith_inferences":["The approach may extend to other model-based reinforcement learning algorithms that already maintain uncertainty estimates.","If the propagation rule scales, similar uncertainty-aware search could reduce the need for separate exploration bonuses in long-horizon sparse-reward domains."],"forward_implications":["AlphaZero paired with EMCTS reaches higher sample efficiency than standard AlphaZero on the SUBLEQ assembly task.","Search with EMCTS solves Deep Sea benchmark variations substantially faster than an otherwise identical method that does not use search to estimate uncertainty.","Baseline AlphaZero and MuZero remain practically unable to solve the same Deep Sea variations that EMCTS handles."],"fun_headline_variants":["Epistemic MCTS propagates uncertainty through search","Uncertainty aware Monte Carlo Tree Search for AlphaZero","EMCTS accounts for epistemic uncertainty in exploration","Propagating limited data uncertainty in tree search","Search integrates epistemic uncertainty for deep exploration"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The learned model's epistemic uncertainty estimates remain accurate enough when propagated through the tree to produce reliable exploration bonuses.","fun_headline_variants_meta":{"raw":{"variants":["Epistemic MCTS propagates uncertainty through search","Uncertainty aware Monte Carlo Tree Search for AlphaZero","EMCTS accounts for epistemic uncertainty in exploration","Propagating limited data uncertainty in tree search","Search integrates epistemic uncertainty for deep exploration"]},"model":"grok-4.3","cost_usd":0.00565,"raw_usage":{"total_tokens":2656,"prompt_tokens":577,"num_sources_used":0,"completion_tokens":66,"cost_in_usd_ticks":56499500,"prompt_tokens_details":{"text_tokens":577,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2013,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":577,"tokens_out":66,"duration_ms":11841,"temperature":1.0,"reasoning_tokens":2013,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-24T10:47:17.636752+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Run EMCTS and a baseline without uncertainty propagation on the same Deep Sea variants; if the baseline solves them at comparable speed or sample count, the claimed benefit of search-based uncertainty handling does not hold.","supporting_citations":[],"review_version":1}