Solving Rubik's Cube Without Tricky Sampling

Siyu Liang; Yicheng Lin

REVIEW 4 major objections 6 minor 15 references

Solving Rubik's Cube Without Tricky Sampling

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a PPO-trained policy, rewarded by a learned cost model, solves randomly scrambled 2x2x2 cubes in over 99.4% of 50,000 test cases without any search at test time.

desk verdict A genuinely new training recipe for the cube is undermined by cost labels that are random-walk counts, not the minimal costs the method defines, so the 99.4% success claim is not supported. read the letter →

arxiv 2411.19583 v1 pith:VM2ZOXYR submitted 2024-11-29 cs.LG cs.AI

Yicheng Lin , Siyu Liang This is my paper · ORCID

classification cs.LGcs.AI

keywords Rubik'scubeReinforcementlearningSparserewardCost-to-goestimationProximalpolicyoptimizationChaseNet2x2x2gradient

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a reinforcement learning pipeline that learns to solve the 2x2x2 Rubik's Cube from fully scrambled states, without the tricky sampling of starting episodes near the solved state. Instead of using sparse binary rewards, a neural network called ChaseNet is first trained to estimate the minimal number of twists between any two states, and its predictions are converted into a dense reward for a PPO policy. The claim is that this reward alone, with no tree search and no near-solved starts, lets the learned policy solve over 99.4 percent of 50,000 scrambled test cubes. If true, it suggests that learned cost models can turn a sparse-reward puzzle into a dense-reward one for policy-gradient methods.

What carries the argument

ChaseNet, a neural network $C_\rho(s,t)$ that estimates the cost (minimal number of twists) between two cube states, together with the logarithmic reward $r_i = -\log_b C_\rho(s_{i+1}, s_g)$. The network is trained in a warmup phase on random scrambles labeled by the number of twists applied, then fine-tuned on the policy's own trajectories; PPO then optimizes the policy against this dense, learned reward. The mechanism works by replacing an all-or-nothing sparse signal with a smooth distance-to-go signal that can be learned from states far from solved.

What would settle it

Measure the Spearman correlation between ChaseNet's predicted cost and the true minimal-twist distance (computed exhaustively for the 2x2x2 cube) on a held-out set of scrambled states; if the correlation is low, the reward no longer tracks progress and the 99.4% success rate should not be reproducible.

Watch

Extended reading notes

Core claim

On the paper's terms, the discovery is that a policy-gradient agent can solve a deliberately scrambled 2x2x2 cube in over 99.4% of 50,000 test cases using only the policy network at test time, provided its reward is shaped by a learned cost model. The cost model ChaseNet is trained first on random scrambles, then fine-tuned on trajectories produced by the very policy being trained, so it keeps tracking the states the agent actually visits. The reward given to PPO is $r_i = -\log_b C_\rho(s_{i+1}, s_g)$ with $b=1.2$, and reaching the solved state gives fixed reward 100; the agent never sees the actual solve distance, only the cost model's estimate. The authors contrast this with earlier deep-RL cube solvers that start episodes from states near the solution and rely on search at test time.

Load-bearing premise

The training labels assume that the number of twists a trajectory has taken equals the true shortest distance between its start and current states; for the fine-tuning stage these labels come from the policy's own imperfect play, so the cost model can inherit the policy's mistakes.

Editorial extensions

If this is right

Other sparse-reward environments could be approached by first learning a cost model between states from random starts, then using its predictions as reward.
Test-time search may be unnecessary in domains where the learned cost model gives enough signal for a policy to act greedily.
The same cost model could be reused as a heuristic for planning, potentially improving sample efficiency further.
Since success is measured with policy alone, the result sets a baseline for how far reward shaping alone can take a policy-gradient agent on combinatorial puzzles.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not claim it, but ChaseNet's cost estimates could be plugged into a search procedure to solve longer or harder scrambles than the policy manages alone.
One extension would be to test whether the learned cost model transfers across scrambble distributions or to different cube sizes, which would show whether it has learned true distance rather than dataset-specific cues.
The fine-tuning label is the episode step index, so the cost model is fit to the current policy's progress; if the policy improves, the labels become better estimates, giving a plausible self-reinforcing loop that the paper leaves implicit.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this paper before citing it: the core idea is fresh—train a pairwise cost model on fully scrambled states and use it as a shaped reward for PPO, no search, no solved-state sampling—but the training labels are not what the paper claims they are. Section 2.2 defines cost as the minimal number of scrambles between states; Algorithm 1 labels each pair (ss, si) with y_i = i, the number of random twists applied so far. A random walk is not a shortest path. Sequences like a then a^-1 return to the same state, and in the 2x2x2 cube the true minimal distance saturates at the diameter while y_i grows linearly. So ChaseNet is regressing toward a quantity that is not the cost defined in §2.2.

Algorithm 2 makes this worse: it fine-tunes ChaseNet with labels equal to the current policy's episode step index. The reward model is fit to the policy's own trajectories, and then that same reward model is used to train that policy. The loop is self-referential. The reward r_i = -log_b C_rho(s_{i+1}, sg) is treated as distance-to-go, but nothing in the described procedure gives ChaseNet ground-truth distances to the solved state for the pairs on which it is evaluated. The learned reward can be self-consistent yet externally ungrounded; the policy may optimize it without making true progress.

What is actually new here is worth credit: training directly from fully scrambled states, without search and without solved-state sampling, is a real departure from DeepCubeA. The paper is clearly written and the FC-vs-attention comparison is a reasonable exploratory step. The authors are honest that they only test on 2x2x2.

But the experimental support is thin beyond that. The success rate is reported with an inconsistent test count (50 attempts in the definition, 50,000 in the abstract), no baselines, no error bars, no episode length limits, and no code. Even the Spearman correlations in §3.1 are reported against "true cost" on a test set, but by the paper's own training procedure, those test labels must also be random-walk counts unless an independent source is introduced—none is described.

My take: the central claim is not supported by the described training loop. The authors should redo the cost labels using an independent distance oracle (for 2x2x2, BFS is trivial), add baselines like a random policy and a simple heuristic, and fix the test-count inconsistency. If they do that, the idea might be worth another look. As it stands, I would not send this to serious peer review; it needs a fundamental revision first.

Referee Report

4 major / 6 minor

Summary. The paper proposes the NX Module, in which a neural network called ChaseNet is trained to predict the cost between state pairs, and a PPO policy is trained using the reward r_i = -log_b C_rho(s_{i+1}, s_g), with a fixed bonus of 100 when the solved state is reached. ChaseNet is first warmed up on random-scramble trajectories (Algorithm 1) and then fine-tuned on trajectories produced by the policy being trained (Algorithm 2). The authors report solving scrambled 2x2x2 Rubik's Cubes in over 99.4% of 50,000 test cases using only the policy network without tree search. Section 4 acknowledges that the method is tested only on the 2x2x2 cube and that scaling remains open.

Significance. If the claimed result were supported, training a policy directly from fully scrambled states without search would be a useful step for sparse-reward RL and would contrast instructively with DeepCubeA's solved-state sampling. The paper also presents two ChaseNet architectures and reports a Spearman correlation analysis. However, the central empirical claim is not currently supported: the reward model is trained on labels that are not the minimal costs defined in Section 2.2, the fine-tuning loop in Algorithm 2 is circular in a way that can unground the reward, and the reported test protocol is internally inconsistent and lacks baselines and confidence intervals. These are load-bearing problems for the main claim rather than presentation issues.

major comments (4)

[§2.3, Algorithm 1] The labels y_i = i in Algorithm 1 are not the cost defined in Section 2.2. The paper defines cost as the minimal number of scrambles between a start state and a target state, but Algorithm 1 labels each pair (s_s, s_i) with the length of the particular random walk that generated the pair. Random-walk lengths are systematically larger than shortest-path distances, can include move sequences that cancel, and can exceed the diameter of the 2x2x2 state graph, while the defined cost cannot. Consequently, ChaseNet is not trained to estimate the quantity C_rho(s_{i+1}, s_g) used in Eq. (2.1), and the reward given to the policy is not grounded in true distance to the solved state. This directly undermines the 99.4% success claim in §3.2.
[§2.5, Algorithm 2] The fine-tuning step D ← D ∪ (s_s, s_i, i) labels ChaseNet pairs with the current policy's own episode step index i. This makes the reward model fit the trajectory lengths of the very policy being trained. If the policy is poor, the labels are poor estimates of minimal cost, and the PPO update can maximize -log_b C_rho(s_{i+1}, s_g) without reducing true distance to s_g. The paper provides no evidence that C_rho, after this fine-tuning, correlates with true minimal cost on (s, s_g) pairs. The training loop is therefore circular with respect to the reward signal, and the reported success rate does not establish that the policy is learning to reach the solved state.
[§3.2] The evaluation protocol is internally inconsistent and under-specified. The text first states that success rate is the ratio over 50 test attempts, while the final sentence claims a success rate of 99.4% across 50,000 test cases. There is no stated episode-length cap, no confidence interval, no number of random seeds, no description of how the test scrambles are generated, and no comparison baseline such as a random policy, a shortest-path solver, or DeepCubeA. The 50-attempt versus 50,000-attempt discrepancy is a load-bearing inconsistency because the headline result is a single aggregate number without a reproducible protocol.
[§3.1 and §3.2] The paper compares ChaseNet-FC and ChaseNet-Attention but does not say which architecture produces the reported 99.4% final success rate. Figure 3.1c plots success rates for both variants during RL training, yet the final validation number is reported only as 'over 99.4% across 50,000 test cases' with no variant attribution. Since the accuracy of the cost model is central to the method, the absence of a variant-specific test protocol for the headline number prevents the reader from assessing which component drives the result.

minor comments (6)

[§2.4] The phrase 'the resolved state' should read 'the solved state' in the sentence introducing s_g.
[Algorithm 2] The line 'F IN ET U N E(C_rho, X, y)' contains a formatting typo, and the comment 'Set current satate' contains a spelling error; these should be corrected.
[§3.2 and Abstract] The number of test cases is inconsistent: the abstract and final sentence of §3.2 say 50,000, while §3.2 also defines the success rate over 50 test attempts; the authors should use one consistent protocol and report it precisely.
[Algorithm 1] The warmup constraint says 'no action is repeated more than three times in a row' but the stated rationale, that repetition returns the cube to a prior state, applies to four consecutive identical moves for a 2x2x2 face turn; the intended constraint should be clarified.
[References] Reference [2] lists the authors in an incorrect and garbled order ('Alexander Shmakov Pierre Baldi Forest Agostinelli, Stephen McAleer') and should be fixed to match the DeepCubeA author list.
[Figure 3.1] The success-rate panel in Figure 3.1c would be more informative with shaded confidence intervals, axis labels, and a statement of how many episodes or test cases each point represents.

Circularity Check

2 steps flagged · score 6.0 of 10

ChaseNet's 'cost' labels are random-walk or policy-trajectory step counts, not the minimal costs defined in §2.2, and the reward is a negative log of this self-referential fit.

fitted input called prediction [Section 2.2, Section 2.3, Algorithm 1, and Eq. (2.1)]
"We define the cost between state pair ( ss, st) as the minimal number of scrambles needed from the start state ss to the target state st. ... Compute the cost yi = i (the number of twists applied) ... D ← D ∪ (ss, si, yi) ... ri = − logb Cρ(si+1, sg) (2.1)"

ChaseNet is fitted to labels yi = i, the number of random twists applied so far, while §2.2 defines cost as the minimal number of scrambles between the pair. A random walk is not a shortest path, so the fitted regression target is not the quantity defined as cost. The paper then uses Cρ(s_{i+1}, s_g) in the reward as if it were the cost to the solved state. The 'predicted cost' used in Eq. (2.1) is therefore, by construction, a fit to a different label renamed as cost, not to the §2.2 cost it is claimed to estimate.
self definitional [Section 2.5, Algorithm 2]
"ri ← −logb Cρ(si+1, sg) ▷ Predict cost using ChaseNet with sg the resolved state ... D ← D ∪ (ss, si, i) ▷ Collect Data for ChaseNet fine tuning ... ρ ← F IN ET U N E(Cρ, X, y) with each Xi = (ss, si) in D"

In Algorithm 2, ChaseNet is fine-tuned on labels i, the episode step index of the very policy being trained, while the same policy is rewarded by −log_b Cρ(s_{i+1}, s_g). Thus the model that defines the reward is updated using labels generated by the policy's own trajectories. The policy and the reward model co-adapt: the reward is a self-consistent function of the policy's step counts rather than an externally grounded minimal cost. The training loop does not connect Cρ to the §2.2 minimal-cost definition, so the claimed cost-to-go signal is defined through the policy's own behavior, making the reward construction self-referential.

full rationale

The paper's headline result—99.4% success on 50,000 test scrambles—is an external benchmark: the policy either reaches the solved state or it does not, so the reported success rate is not itself forced by the training definitions. That external check prevents a score of 8 or 10. However, the claimed mechanism is not a first-principles derivation. The paper defines cost as minimal scramble count, but Algorithm 1 labels ChaseNet's training data with random-walk twist counts, and Algorithm 2 fine-tunes the same model with the current policy's episode step index. The reward in Eq. (2.1) then treats Cρ(s_{i+1}, s_g) as a cost-to-go, even though the training targets are not minimal costs and the fine-tuning targets are generated by the policy being optimized. This is a fitted proxy called a cost prediction, and the fine-tuning loop is self-referential. No uniqueness theorem or self-citation chain is involved. Because the central 'cost prediction' reduces, by construction, to fitting a different label, while the solve rate remains an independent external metric, partial circularity is the appropriate finding.

Assumptions & free parameters 2 free parameters · 2 assumptions · 0 invented entities

The central claim rests on a hand-chosen reward base, a data-generation restriction, and two unproven assumptions about cost labels and reward shaping. No new physical or conceptual entities are introduced.

free parameters (2)

b (reward base) = 1.2
Hand-chosen base in the reward r_i = -log_b C_rho(s_{i+1}, s_g); affects reward scale and therefore policy optimization.
Max repetitions of an action in warmup = 3
Data generation rule in Section 2.3 that restricts no action from being repeated more than three times in a row; a hand-chosen threshold that biases the cost labels.

assumptions (2)

domain assumption The number of random twists i equals the true cost between states s_s and s_i
Algorithm 1 uses y_i = i as the label for ChaseNet, but random scrambles are not guaranteed to be optimal, so the labels are upper bounds on true cost.
domain assumption The reward -log_b C_rho(s_{i+1}, s_g) preserves the optimal policy under PPO
The paper does not prove reward shaping preserves the optimal policy for this cost model; the shaped reward may induce suboptimal behavior.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Solving Rubik's Cube Without Tricky Sampling." pith.science (2026). https://pith.science/paper/VM2ZOXYR

@misc{pith2026241119583,
  author       = {Pith},
  title        = {Pith review of: Solving Rubik's Cube Without Tricky Sampling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VM2ZOXYR}},
  note         = {Machine review of arXiv:2411.19583}
}

read the original abstract

The Rubiks Cube, with its vast state space and sparse reward structure, presents a significant challenge for reinforcement learning (RL) due to the difficulty of reaching rewarded states. Previous research addressed this by propagating cost-to-go estimates from the solved state and incorporating search techniques. These approaches differ from human strategies that start from fully scrambled cubes, which can be tricky for solving a general sparse-reward problem. In this paper, we introduce a novel RL algorithm using policy gradient methods to solve the Rubiks Cube without relying on near solved-state sampling. Our approach employs a neural network to predict cost patterns between states, allowing the agent to learn directly from scrambled states. Our method was tested on the 2x2x2 Rubiks Cube, where the cube was scrambled 50,000 times, and the model successfully solved it in over 99.4% of cases. Notably, this result was achieved using only the policy network without relying on tree search as in previous methods, demonstrating its effectiveness and potential for broader applications in sparse-reward problems.

Figures

Figures reproduced from arXiv: 2411.19583 by the authors.

**Figure 2.1.** General Representation of Rubik’s cube problem. a. The state of the cube can be [PITH_FULL_IMAGE:figures/full_fig_p003_2_1.png] view at source ↗

**Figure 2.2.** The NX Module a. The NX Module training process is divided into a warmup phase and [PITH_FULL_IMAGE:figures/full_fig_p003_2_2.png] view at source ↗

**Figure 3.1.** a. Warmup loss for ChaseNet-FC and ChaseNet-Attention. b. Average rewards during [PITH_FULL_IMAGE:figures/full_fig_p007_3_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 10 canonical work pages

[1]

Towards learning rubik’s cube with n-tuple-based reinforcement learning, 2023

Wolfgang Konen. Towards learning rubik’s cube with n-tuple-based reinforcement learning, 2023

work page 2023
[2]

Solving the rubik’s cube with deep reinforcement learning and search

Alexander Shmakov Pierre Baldi Forest Agostinelli, Stephen McAleer. Solving the rubik’s cube with deep reinforcement learning and search. nature machine intelligence, 1:356–363, 2019

work page 2019
[3]

Solving the rubik’s cube with approximate policy iteration

Stephen McAleer, Forest Agostinelli, Alexander Shmakov, and Pierre Baldi. Solving the rubik’s cube with approximate policy iteration. In International Conference on Learning Representations, 2019

work page 2019
[4]

Rupam Mahmood

Gautham Vasan, Yan Wang, Fahim Shahriar, James Bergstra, Martin Jagersand, and A. Rupam Mahmood. Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learning. arXiv e-prints, page arXiv:2407.00324, June 2024

arXiv 2024
[5]

Learning by Playing - Solving Sparse Reward Tasks from Scratch

Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by Playing - Solving Sparse Reward Tasks from Scratch. arXiv e-prints, page arXiv:1802.10567, February 2018

arXiv 2018
[6]

Hindsight Experience Replay

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight Experience Replay. arXiv e-prints, page arXiv:1707.01495, July 2017

arXiv 2017
[7]

The diameter of the rubik’s cube group is twenty

Tomas Rokicki, Herbert Kociemba, Morley Davidson, and John Dethridge. The diameter of the rubik’s cube group is twenty. SIAM Review, 56(4):645–670, 2014

work page 2014
[8]

Bertsekas and J.N

D.P. Bertsekas and J.N. Tsitsiklis. Neuro-dynamic programming: an overview. In Proceedings of 1995 34th IEEE Conference on Decision and Control, volume 1, pages 560–564 vol.1, 1995

work page 1995

Show all 15 references

[9]

Bagnell, Sham M Kakade, Jeff Schneider, and Andrew Ng

J. Bagnell, Sham M Kakade, Jeff Schneider, and Andrew Ng. Policy search by dynamic programming. In S. Thrun, L. Saul, and B. Sch¨ olkopf, editors, Advances in Neural Information Processing Systems, volume 16. MIT Press, 2003

2003
[10]

Kakade and John Langford

Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning, 2002

2002
[11]

Approximate Policy Iteration Schemes: A Comparison

Bruno Scherrer. Approximate Policy Iteration Schemes: A Comparison. arXiv e-prints, page arXiv:1405.2878, May 2014

2014 arXiv
[12]

Analysis of classification-based policy iteration algorithms

Alessandro Lazaric, Mohammad Ghavamzadeh, and R´ emi Munos. Analysis of classification-based policy iteration algorithms. Journal of Machine Learning Research, 17(19):1–30, 2016

2016
[13]

Puterman and Moon Chirl Shin

Martin L. Puterman and Moon Chirl Shin. Modified policy iteration algorithms for discounted markov decision problems. Management Science, 24(11):1127–1137, 1978

1978
[14]

Proximal Policy Optimization Algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv e-prints, page arXiv:1707.06347, July 2017

2017 arXiv
[15]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. arXiv e-prints, page arXiv:1706.03762, June 2017

2017 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Towards learning rubik’s cube with n-tuple-based reinforcement learning, 2023

Wolfgang Konen. Towards learning rubik’s cube with n-tuple-based reinforcement learning, 2023

work page 2023

[2] [2]

Solving the rubik’s cube with deep reinforcement learning and search

Alexander Shmakov Pierre Baldi Forest Agostinelli, Stephen McAleer. Solving the rubik’s cube with deep reinforcement learning and search. nature machine intelligence, 1:356–363, 2019

work page 2019

[3] [3]

Solving the rubik’s cube with approximate policy iteration

Stephen McAleer, Forest Agostinelli, Alexander Shmakov, and Pierre Baldi. Solving the rubik’s cube with approximate policy iteration. In International Conference on Learning Representations, 2019

work page 2019

[4] [4]

Rupam Mahmood

Gautham Vasan, Yan Wang, Fahim Shahriar, James Bergstra, Martin Jagersand, and A. Rupam Mahmood. Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learning. arXiv e-prints, page arXiv:2407.00324, June 2024

arXiv 2024

[5] [5]

Learning by Playing - Solving Sparse Reward Tasks from Scratch

Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by Playing - Solving Sparse Reward Tasks from Scratch. arXiv e-prints, page arXiv:1802.10567, February 2018

arXiv 2018

[6] [6]

Hindsight Experience Replay

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight Experience Replay. arXiv e-prints, page arXiv:1707.01495, July 2017

arXiv 2017

[7] [7]

The diameter of the rubik’s cube group is twenty

Tomas Rokicki, Herbert Kociemba, Morley Davidson, and John Dethridge. The diameter of the rubik’s cube group is twenty. SIAM Review, 56(4):645–670, 2014

work page 2014

[8] [8]

Bertsekas and J.N

D.P. Bertsekas and J.N. Tsitsiklis. Neuro-dynamic programming: an overview. In Proceedings of 1995 34th IEEE Conference on Decision and Control, volume 1, pages 560–564 vol.1, 1995

work page 1995