{"id":"3f9466e1-51c6-46fc-a6ab-62a6f67a1724","arxiv_id":"2411.19583","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper presents a PPO policy trained with rewards from a learned cost model, claiming 99.4% success on the 2x2x2 Rubik's Cube without search or solved-state sampling, but with weak evidential support.","lead":"A reinforcement learning agent solves the 2x2x2 Rubik's Cube from fully scrambled states by first learning a cost model, then using that model as a reward signal, with no search at test time. The paper reports a 99.4% success rate, but the evidence is weakened by a self-referential training loop and missing baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ChaseNet's training labels are random-walk counts, not the minimal costs defined in §2.2, so the PPO reward is not grounded in true distance to the goal.","rationale":"The reader's weakest_assumption correctly identifies Algorithm 2's policy-generated labels as a circularity, but the deeper problem is broader: even the warmup labels in Algorithm 1 violate the paper's own definition of cost. Because y_i is a random-walk length rather than a shortest-path length, it can exceed the true cost arbitrarily (up to the point where true cost saturates at the diameter), and it can be far above zero for states that are identical or close. This is not a disagreement with an external consensus; it is an internal inconsistency between §2.2 and the two training algorithms. The absence of goal-target pairs in the ChaseNet training data compounds the issue, since the model is deployed on inputs of the form (s, sg) though it is trained on pairs of arbitrary scrambled states. These issues are load-bearing because the only training signal for the policy is the ChaseNet-based reward; if that reward is not a grounded cost estimate, the reported success rate can reflect optimization of a self-consistent but arbitrary potential rather than actual progress. The proposed BFS-based relabeling experiment would settle the concern directly: exact labels either preserve the result or reveal that it depended on the biased labels. I agree with the reader's REJECT verdict, so no adjustment is needed.","tokens_in":5880,"tokens_out":5955,"duration_ms":60287,"concrete_test":"Generate a sample of state pairs exactly as in Algorithm 1, plus a second sample from Algorithm 2's dataset D, and compute the true minimal cost for each pair by bidirectional BFS over the full 3,674,160-state 2x2x2 graph. Report the distribution of y_i - true_cost and the Spearman correlation of ChaseNet's predictions against true cost on goal-target pairs. Then retrain the full pipeline with y_i replaced by the exact minimal cost, keeping all hyperparameters identical, and rerun the 50,000-scramble evaluation. If the >99.4% success rate survives exact labels, the concern does not land; if it collapses or requires tree search, the reported result depended on the non-cost, self-referential labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim hinges on ChaseNet supplying a reliable cost-to-go signal, but the only labels used in training are not costs. Section 2.2 defines cost as the minimal number of scrambles between states. Algorithm 1 labels each pair (ss, si) with y_i = i, the number of random twists applied so far. A random walk is not a shortest path: in the 2x2x2 cube, sequences like a then a^{-1} return to the same state, and more generally the true minimal distance is usually far below i and saturates at the cube's diameter, while y_i grows linearly. Thus ChaseNet is trained by regression to predict a quantity that is not the quantity defined in §2.2. Algorithm 2 makes this worse by fine-tuning ChaseNet with labels equal to the current policy's episode step index, so the reward model is fit to the policy's own trajectory lengths. The reward in Eq. 2.1 then treats C_rho(s_{i+1}, sg) as if it were distance to the solved state, but neither algorithm systematically provides training pairs with sg as the target state. The learned reward can therefore be a self-consistent but externally ungrounded measure; the policy may optimize it without making true progress toward the goal. Without independent distance labels or a demonstration that ChaseNet correlates with true cost on goal-target pairs, the 99.4% success claim is not supported by the described training loop.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the NX Module, in which a neural network called ChaseNet is trained to predict the cost between state pairs, and a PPO policy is trained using the reward r_i = -log_b C_rho(s_{i+1}, s_g), with a fixed bonus of 100 when the solved state is reached. ChaseNet is first warmed up on random-scramble trajectories (Algorithm 1) and then fine-tuned on trajectories produced by the policy being trained (Algorithm 2). The authors report solving scrambled 2x2x2 Rubik's Cubes in over 99.4% of 50,000 test cases using only the policy network without tree search. Section 4 acknowledges that the method is tested only on the 2x2x2 cube and that scaling remains open.","tokens_in":6177,"tokens_out":5450,"duration_ms":50036,"significance":"If the claimed result were supported, training a policy directly from fully scrambled states without search would be a useful step for sparse-reward RL and would contrast instructively with DeepCubeA's solved-state sampling. The paper also presents two ChaseNet architectures and reports a Spearman correlation analysis. However, the central empirical claim is not currently supported: the reward model is trained on labels that are not the minimal costs defined in Section 2.2, the fine-tuning loop in Algorithm 2 is circular in a way that can unground the reward, and the reported test protocol is internally inconsistent and lacks baselines and confidence intervals. These are load-bearing problems for the main claim rather than presentation issues.","major_comments":[{"comment":"The labels y_i = i in Algorithm 1 are not the cost defined in Section 2.2. The paper defines cost as the minimal number of scrambles between a start state and a target state, but Algorithm 1 labels each pair (s_s, s_i) with the length of the particular random walk that generated the pair. Random-walk lengths are systematically larger than shortest-path distances, can include move sequences that cancel, and can exceed the diameter of the 2x2x2 state graph, while the defined cost cannot. Consequently, ChaseNet is not trained to estimate the quantity C_rho(s_{i+1}, s_g) used in Eq. (2.1), and the reward given to the policy is not grounded in true distance to the solved state. This directly undermines the 99.4% success claim in §3.2.","section":"§2.3, Algorithm 1"},{"comment":"The fine-tuning step D ← D ∪ (s_s, s_i, i) labels ChaseNet pairs with the current policy's own episode step index i. This makes the reward model fit the trajectory lengths of the very policy being trained. If the policy is poor, the labels are poor estimates of minimal cost, and the PPO update can maximize -log_b C_rho(s_{i+1}, s_g) without reducing true distance to s_g. The paper provides no evidence that C_rho, after this fine-tuning, correlates with true minimal cost on (s, s_g) pairs. The training loop is therefore circular with respect to the reward signal, and the reported success rate does not establish that the policy is learning to reach the solved state.","section":"§2.5, Algorithm 2"},{"comment":"The evaluation protocol is internally inconsistent and under-specified. The text first states that success rate is the ratio over 50 test attempts, while the final sentence claims a success rate of 99.4% across 50,000 test cases. There is no stated episode-length cap, no confidence interval, no number of random seeds, no description of how the test scrambles are generated, and no comparison baseline such as a random policy, a shortest-path solver, or DeepCubeA. The 50-attempt versus 50,000-attempt discrepancy is a load-bearing inconsistency because the headline result is a single aggregate number without a reproducible protocol.","section":"§3.2"},{"comment":"The paper compares ChaseNet-FC and ChaseNet-Attention but does not say which architecture produces the reported 99.4% final success rate. Figure 3.1c plots success rates for both variants during RL training, yet the final validation number is reported only as 'over 99.4% across 50,000 test cases' with no variant attribution. Since the accuracy of the cost model is central to the method, the absence of a variant-specific test protocol for the headline number prevents the reader from assessing which component drives the result.","section":"§3.1 and §3.2"}],"minor_comments":[{"comment":"The phrase 'the resolved state' should read 'the solved state' in the sentence introducing s_g.","section":"§2.4"},{"comment":"The line 'F IN ET U N E(C_rho, X, y)' contains a formatting typo, and the comment 'Set current satate' contains a spelling error; these should be corrected.","section":"Algorithm 2"},{"comment":"The number of test cases is inconsistent: the abstract and final sentence of §3.2 say 50,000, while §3.2 also defines the success rate over 50 test attempts; the authors should use one consistent protocol and report it precisely.","section":"§3.2 and Abstract"},{"comment":"The warmup constraint says 'no action is repeated more than three times in a row' but the stated rationale, that repetition returns the cube to a prior state, applies to four consecutive identical moves for a 2x2x2 face turn; the intended constraint should be clarified.","section":"Algorithm 1"},{"comment":"Reference [2] lists the authors in an incorrect and garbled order ('Alexander Shmakov Pierre Baldi Forest Agostinelli, Stephen McAleer') and should be fixed to match the DeepCubeA author list.","section":"References"},{"comment":"The success-rate panel in Figure 3.1c would be more informative with shaded confidence intervals, axis labels, and a statement of how many episodes or test cases each point represents.","section":"Figure 3.1"}],"recommendation":"reject","confidential_remarks":"I concur with the reader's assessment that the reward model is not grounded in the defined cost and that the fine-tuning loop is circular. The inconsistencies in the evaluation protocol (50 vs. 50,000 test cases, no baseline, no episode cap) make the central claim unverifiable as reported. A future resubmission with true-distance labels, independent cost-model validation, and a complete evaluation protocol would be needed before the claim can be assessed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper before citing it: the core idea is fresh—train a pairwise cost model on fully scrambled states and use it as a shaped reward for PPO, no search, no solved-state sampling—but the training labels are not what the paper claims they are. Section 2.2 defines cost as the minimal number of scrambles between states; Algorithm 1 labels each pair (ss, si) with y_i = i, the number of random twists applied so far. A random walk is not a shortest path. Sequences like a then a^-1 return to the same state, and in the 2x2x2 cube the true minimal distance saturates at the diameter while y_i grows linearly. So ChaseNet is regressing toward a quantity that is not the cost defined in §2.2.\n\nAlgorithm 2 makes this worse: it fine-tunes ChaseNet with labels equal to the current policy's episode step index. The reward model is fit to the policy's own trajectories, and then that same reward model is used to train that policy. The loop is self-referential. The reward r_i = -log_b C_rho(s_{i+1}, sg) is treated as distance-to-go, but nothing in the described procedure gives ChaseNet ground-truth distances to the solved state for the pairs on which it is evaluated. The learned reward can be self-consistent yet externally ungrounded; the policy may optimize it without making true progress.\n\nWhat is actually new here is worth credit: training directly from fully scrambled states, without search and without solved-state sampling, is a real departure from DeepCubeA. The paper is clearly written and the FC-vs-attention comparison is a reasonable exploratory step. The authors are honest that they only test on 2x2x2.\n\nBut the experimental support is thin beyond that. The success rate is reported with an inconsistent test count (50 attempts in the definition, 50,000 in the abstract), no baselines, no error bars, no episode length limits, and no code. Even the Spearman correlations in §3.1 are reported against \"true cost\" on a test set, but by the paper's own training procedure, those test labels must also be random-walk counts unless an independent source is introduced—none is described.\n\nMy take: the central claim is not supported by the described training loop. The authors should redo the cost labels using an independent distance oracle (for 2x2x2, BFS is trivial), add baselines like a random policy and a simple heuristic, and fix the test-count inconsistency. If they do that, the idea might be worth another look. As it stands, I would not send this to serious peer review; it needs a fundamental revision first.","headline":"A genuinely new training recipe for the cube is undermined by cost labels that are random-walk counts, not the minimal costs the method defines, so the 99.4% success claim is not supported.","tokens_in":6701,"tokens_out":1840,"would_cite":false,"duration_ms":17054,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a PPO-trained policy, rewarded by a learned cost model, solves randomly scrambled 2x2x2 cubes in over 99.4% of 50,000 test cases without any search at test time.","keywords":["Rubik's cube","Reinforcement learning","Sparse reward","Cost-to-go estimation","Proximal policy optimization","ChaseNet","2x2x2 cube","Policy gradient"],"falsifier":"Measure the Spearman correlation between ChaseNet's predicted cost and the true minimal-twist distance (computed exhaustively for the 2x2x2 cube) on a held-out set of scrambled states; if the correlation is low, the reward no longer tracks progress and the 99.4% success rate should not be reproducible.","tokens_in":5684,"feed_emoji":"🧩","tokens_out":4802,"duration_ms":39606,"temperature":0.7,"pith_summary":"This paper proposes a reinforcement learning pipeline that learns to solve the 2x2x2 Rubik's Cube from fully scrambled states, without the tricky sampling of starting episodes near the solved state. Instead of using sparse binary rewards, a neural network called ChaseNet is first trained to estimate the minimal number of twists between any two states, and its predictions are converted into a dense reward for a PPO policy. The claim is that this reward alone, with no tree search and no near-solved starts, lets the learned policy solve over 99.4 percent of 50,000 scrambled test cubes. If true, it suggests that learned cost models can turn a sparse-reward puzzle into a dense-reward one for policy-gradient methods.","feed_headline":"Policy-only RL solves scrambled 2x2x2 cubes 99.4%","feed_subtitle":"A learned cost model turns sparse rewards dense, so the agent learns directly from fully scrambled states.","key_machinery":"ChaseNet, a neural network $C_\\rho(s,t)$ that estimates the cost (minimal number of twists) between two cube states, together with the logarithmic reward $r_i = -\\log_b C_\\rho(s_{i+1}, s_g)$. The network is trained in a warmup phase on random scrambles labeled by the number of twists applied, then fine-tuned on the policy's own trajectories; PPO then optimizes the policy against this dense, learned reward. The mechanism works by replacing an all-or-nothing sparse signal with a smooth distance-to-go signal that can be learned from states far from solved.","core_discovery":"On the paper's terms, the discovery is that a policy-gradient agent can solve a deliberately scrambled 2x2x2 cube in over 99.4% of 50,000 test cases using only the policy network at test time, provided its reward is shaped by a learned cost model. The cost model ChaseNet is trained first on random scrambles, then fine-tuned on trajectories produced by the very policy being trained, so it keeps tracking the states the agent actually visits. The reward given to PPO is $r_i = -\\log_b C_\\rho(s_{i+1}, s_g)$ with $b=1.2$, and reaching the solved state gives fixed reward 100; the agent never sees the actual solve distance, only the cost model's estimate. The authors contrast this with earlier deep-RL cube solvers that start episodes from states near the solution and rely on search at test time.","pith_inferences":["The paper does not claim it, but ChaseNet's cost estimates could be plugged into a search procedure to solve longer or harder scrambles than the policy manages alone.","One extension would be to test whether the learned cost model transfers across scrambble distributions or to different cube sizes, which would show whether it has learned true distance rather than dataset-specific cues.","The fine-tuning label is the episode step index, so the cost model is fit to the current policy's progress; if the policy improves, the labels become better estimates, giving a plausible self-reinforcing loop that the paper leaves implicit."],"forward_implications":["Other sparse-reward environments could be approached by first learning a cost model between states from random starts, then using its predictions as reward.","Test-time search may be unnecessary in domains where the learned cost model gives enough signal for a policy to act greedily.","The same cost model could be reused as a heuristic for planning, potentially improving sample efficiency further.","Since success is measured with policy alone, the result sets a baseline for how far reward shaping alone can take a policy-gradient agent on combinatorial puzzles."],"supporting_citations":[{"why":"the prior deep-RL solver that samples states near the solved state and uses search, the approach this paper's method is designed to avoid.","marker":"[2]"},{"why":"the earlier deep-RL cube solver that this work contrasts with on sampling strategy and test-time search.","marker":"[3]"},{"why":"the Proximal Policy Optimization algorithm used to train the policy network.","marker":"[14]"},{"why":"the transformer architecture that ChaseNet-Attention builds on to predict state-pair costs.","marker":"[15]"}],"fun_headline_variants":["No search, no near-solved states: RL cracks Rubik's 99.4%","Policy-only RL beats sparse rewards: solves scrambled Rubik's 99.4%","Learned cost model turns sparse dense, RL solves Rubik's 99.4%","Rubik's without tricky sampling: policy-only RL hits 99.4%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training labels assume that the number of twists a trajectory has taken equals the true shortest distance between its start and current states; for the fine-tuning stage these labels come from the policy's own imperfect play, so the cost model can inherit the policy's mistakes.","fun_headline_variants_meta":{"raw":{"variants":["No search, no near-solved states: RL cracks Rubik's 99.4%","Policy-only RL beats sparse rewards: solves scrambled Rubik's 99.4%","Learned cost model turns sparse dense, RL solves Rubik's 99.4%","Rubik's without tricky sampling: policy-only RL hits 99.4%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000395,"raw_usage":{"total_tokens":2049,"prompt_tokens":903,"completion_tokens":1146,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":1053}},"tokens_in":519,"tokens_out":1146,"duration_ms":9519,"temperature":1.0,"reasoning_tokens":1053,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:01:40.472639+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the Spearman correlation between ChaseNet's predicted cost and the true minimal-twist distance (computed exhaustively for the 2x2x2 cube) on a held-out set of scrambled states; if the correlation is low, the reward no longer tracks progress and the 99.4% success rate should not be reproducible.","supporting_citations":[{"cited_title":"Solving the rubik’s cube with deep reinforcement learning and search","cited_arxiv_id":null,"evidence_quote":"the prior deep-RL solver that samples states near the solved state and uses search, the approach this paper's method is designed to avoid."},{"cited_title":"Solving the rubik’s cube with approximate policy iteration","cited_arxiv_id":null,"evidence_quote":"the earlier deep-RL cube solver that this work contrasts with on sampling strategy and test-time search."}],"review_version":1}