Mapping State Space using Landmarks for Universal Goal Reaching

Fangchen Liu; Hao Su; Zhiao Huang

REVIEW 3 major objections 7 minor 37 references

Mapping State Space using Landmarks for Universal Goal Reaching

T0 review · 3 major / 7 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A landmark map turns local value estimates into long-range goal reaching in RL.

desk verdict A plausible landmark-graph planner for goal-conditioned RL that gets the core idea right, but the evaluation doesn't yet quantify the load-bearing assumption that local UVFA estimates are reliable within the clipping threshold. read the letter →

arxiv 1908.05451 v1 pith:TA2I6QQH submitted 2019-08-15 cs.LG stat.ML

Zhiao Huang , Fangchen Liu , Hao Su This is my paper · ORCID

classification cs.LGstat.ML

keywords universalvaluefunctionapproximationgoal-conditionedreinforcementlearninglandmark-basedmapfarthestpointsamplingsparserewardsshortestpathplanninghierarchicalRLlong-rangegoals

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that for goal-reaching tasks in large, sparse-reward environments, a single learned value network cannot reliably estimate distances to faraway goals. Instead, the agent should maintain a small graph map of the visited state space: nodes are landmark states sampled by farthest-point sampling, and edges connect nearby landmarks with distances estimated by a locally accurate value function. A shortest-path search over this map propagates trustable local estimates across long horizons, so the agent can reach distant goals early in training and outperform standard goal-conditioned reinforcement learning. If correct, long-range value estimation can be factored into local learning plus global graph search, reducing sample complexity and dependence on network extrapolation.

What carries the argument

The central object is the landmark-based map: a directed weighted graph whose nodes are states selected from the replay buffer via farthest-point sampling (FPS), and whose edges are UVFA distance estimates between nearby landmarks. FPS iteratively picks the state farthest from already chosen landmarks, spreading landmarks toward the boundary of the visited space and improving exploration. The map does the work of global value propagation: the local UVFA supplies trustworthy edge weights and local control, while the graph supplies topology and long-range routing.

What would settle it

In a maze with known ground-truth distances, compute shortest-path distances over the landmark graph using locally verified UVFA edge weights and compare them to ground-truth distances for all goal pairs; if the planned length is systematically wrong even when every edge's local estimate is correct, the central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that the all-pairs value estimation problem in universal goal reaching can be coarsened into a landmark graph without losing routing accuracy, provided the universal value function approximator is accurate only in a local neighborhood. The paper builds a directed weighted graph from replay-buffer states; it connects landmark pairs only when the UVFA-estimated distance is below a threshold $\tau$, then runs Bellman-Ford to obtain pairwise landmark distances. The next subgoal is chosen by minimizing $d(s, \ell_i) + d(\ell_i, g)$, so the long-range distance estimate is a piecewise shortest path over trusted local estimates rather than a single network extrapolation. Empirically, this planner gives lower mean distortion error on value estimates and higher success rates on maze and manipulation tasks, especially in the early training stage.

Load-bearing premise

The graph is only as trustworthy as the local UVFA distance estimates; if those estimates are biased within the threshold $\tau$, every edge inherits the bias and the planned shortest path is wrong.

Editorial extensions

If this is right

Long-horizon goals in sparse-reward environments become reachable early in training because planning decouples local control from global routing.
A goal-conditioned value network only needs local accuracy, reducing the sample complexity and network capacity required compared with a global UVFA.
Exploration improves because FPS landmarks tend to lie at the frontier of experienced states, guiding the agent outward.
The planner combines with HER-trained DQN and DDPG and yields stable success on long-horizon mazes where standard HER fails.
The graph-based distance estimator naturally extends to state-goal pairs never experienced together, as long as a path through landmarks exists.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same coarsening idea could apply to any learned local distance or reachability signal, not just UVFA, provided edge reliability can be certified.
A testable extension is to grow the edge threshold $\tau$ online as the local value estimates become more accurate, potentially removing the hand-tuned clipping parameter.
If landmark edge weights are consistently biased, shortest-path distances may still be accurate only up to an additive bias; evaluating planned lengths against ground-truth shortest paths, rather than only success rates, would expose this.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know upfront. First, the core idea is genuinely useful: instead of forcing a single UVFA to memorize long-range values, build a dynamic graph from replay states chosen by farthest-point sampling, connect only nearby landmarks with learned local distance estimates, and run shortest-path search. That is a clean way to factor long-horizon goal reaching. Second, the paper is honest about its own limits, but the empirics don't yet back the central claim about local reliability.

What's new is the assembly: FPS plus UVFA-weighted edges plus Bellman-Ford. Individually these are known, but the dynamic construction is a reasonable contribution. The FourRoom experiment is a real strength — validating against ground-truth shortest paths rather than just reporting success rates is the right kind of check, and the early MDE improvement is credible. The related work is also careful: the paper correctly distinguishes itself from PRM-RL (which needs a physical engine) and from Savinov et al. (which needs a supervised connectivity classifier). The FPS-vs-uniform ablation shows a concrete exploration benefit.

The weak spots are real but not fatal. The curves are single-run with no error bars, and hyperparameters like the clip threshold, landmark count, and goal-distance delta are only partially reported. I would like to see comparisons to PRM-RL and Savinov, since those are the closest graph-planning baselines and the paper discusses them without testing. The deeper issue is the threshold tau. The same learned Q supplies both the edge weights and the criterion for whether a pair of landmarks is close enough to trust. If Q underestimates distance, you get false shortcuts; if it overestimates, the graph fragments. The FourRoom MDE in Fig. 2b is averaged over all pairs, not conditioned on connected edges, so it doesn't tell us whether the threshold actually selects reliable local estimates. Section 7's concession about accumulated error confirms the dependency without bounding it. This is a placeholder for future work, not a contradiction of the method, but it should be quantified for the central claim to land.

Overall, the paper is for researchers in goal-conditioned RL and model-based planning who want a simple sample-based way to extend locally accurate value functions to long horizons. It is worth a serious referee. A good reviewer should ask for multi-seed runs, a hyperparameter table, and some diagnostic about edge reliability under varying tau. My recommendation: send to peer review, but expect revision.

Referee Report

3 major / 7 minor

Summary. This paper addresses universal goal reaching in sparse-reward MDPs whose state space is a low-dimensional manifold. The proposed method, 'Planning with State-space Mapping' (Algorithm 1), trains a goal-conditioned Q-function with HER, samples landmark states from the replay buffer using farthest-point sampling, connects landmark pairs whose learned Q-distance falls below a clip threshold to form a directed weighted graph, runs Bellman-Ford to compute pairwise landmark distances, and at execution time selects the next landmark that minimizes the sum of the local value from the current state to a landmark and the planned distance from that landmark to the goal. The paper reports lower mean distortion error than HER on a FourRoom grid with ground-truth shortest paths, higher success rates on 2DReach, 2DPush, PointMaze, AntMaze, Complex AntMaze, Acrobot, and BlockedFetchReach/FetchPush, and an advantage over HIRO on a large AntMaze in a table. Ablations study the clip threshold, the number of landmarks, FPS vs uniform sampling, and local accuracy of the learned value function.

Significance. If the empirical claims hold, the paper makes a useful conceptual contribution: it decomposes long-horizon value estimation into a graph-based global planner over landmark nodes and a locally trained UVFA, thereby avoiding extrapolation over long distances. The FourRoom experiment (Fig. 2) is a genuine validation because mean distortion error is measured against ground-truth shortest paths. The method is simple and the Algorithm 1 pseudocode is clear. However, the evidence is not yet at the level needed to support the central claim: the key reliability assumption about local edge weights is not directly tested, and the continuous-control comparisons lack seeds, error bars, and full hyperparameter reporting. The paper is therefore promising but requires additional experiments and diagnostics before publication.

major comments (3)

[Section 5.3 / Algorithm 1, lines 4-8] The graph's edge set is determined by the same learned Q that supplies the edge weights: an edge (v_i, v_j) is included iff min_a -Q(v_i, v_j, a) <= clip_bound. The manuscript asserts that 'UVFA is accurate locally but unreliable for long-term future' (Sec. 5.3) but never measures accuracy conditioned on the pairs that are actually connected in the graph. Fig. 2b's mean distortion error is averaged over all state-goal pairs, not over landmark pairs that pass the threshold; no analogous calibration is reported for the continuous environments. Section 6.3 reports only end success rates for a few tau values. This is load-bearing because if Q underestimates a genuinely long-range pair, that pair enters the graph as a false shortcut, and if it overestimates, the graph fragments. Please add a diagnostic (e.g., in FourRoom, compare UVFA edge weights with ground-truth distances for threshold-passing landmark pairs; in continuous environments, report edge-connectivity statistics and the sensitivity of success to tau) and condition the MDE on graph edges.
[Section 4, Eq. (1) and Algorithm 1] The conversion V ≈ -d rests on the bound T^2 Rmax(1-gamma) -> 0. The paper does not report gamma, Rmax, or T for the experiments. For AntMaze with horizon 500 and gamma=0.99, T^2 Rmax(1-gamma) = 2500 Rmax, which is not close to zero. This does not necessarily invalidate the method, but it means the 'distances' used as edge weights and planned over are not justified quantitatively by the stated derivation. Please either state the actual gamma and horizon values, replace Eq. (1) with an episodic undiscounted formulation, or provide a bound that is meaningful in the experimental regime.
[Section 6 / Table 1] Figure 4 and Table 1 report success-rate curves and values without error bars or the number of seeds. Table 1 is introduced as a comparison with HIRO and HAC, but HAC rows do not appear, and the text says HAC 'couldn't make it work well' without giving numbers. Hyperparameters such as the number of landmarks, clip_bound/tau, goal-reaching delta, HER relabeling horizon, network architecture, learning rate, and replay buffer size are not stated. Without these, the central empirical claim that the planner 'achieve better performance than standard RL algorithms' (Abstract) cannot be independently evaluated. Please add multi-seed mean/std, a full hyperparameter table, and the missing HAC results, or remove HAC from the comparison.

minor comments (7)

[Section 5.3] Algorithm 1 is referred to as 'Algorithm 10' in the text; correct the reference.
[Section 6.2.2] The text says '2DReach and 2DPush ... shown in Figure 4b', but 2DReach appears in Figure 4a.
[Figure 2b] The legend contains 'uni00A0' artifacts; clean up the labels.
[Figure 4] The caption says the red curve indicates the performance of our method, but the curves in the figure are not visually distinguished in grayscale; use distinct markers or colors and a legend.
[Section 4] The notation 'T2Rmax' should be 'T^2 Rmax'; also clarify the role of epsilon in the Taylor expansion.
[Section 5.2] The modification of HER 'ensuring that the replaced goals are sampled from the near future within a fixed number of steps' is never quantified; state the fixed number.
[Table 1] In the 'Ours Sparse' row, the success rate reaches 0.5 at 0.5M steps and remains 0.5 at 1.5M and 2M; clarify whether this is a plateau or a reporting artifact.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the landmark map is a compositional use of learned local estimates, and the paper's headline claims are validated against external success rates and ground-truth distances.

full rationale

The paper's derivation chain is: train a local UVFA with HER, sample landmarks from the replay buffer using FPS, connect only landmark pairs whose UVFA-estimated distance is below a threshold, run Bellman-Ford on that graph, and output the next subgoal. The claimed benefit is that shortest paths over locally accurate edges give better long-range value estimates than a single global UVFA. Nothing in this chain is equivalent to its inputs by construction. The edge weights are the same learned Q estimates used to decide connectivity, but this is a compositional design choice, not a fitted parameter renamed as a prediction; the FourRoom experiments directly validate the estimated distances against ground-truth shortest-path distances via mean distortion error (Fig. 2b) and evaluate success rates externally. The threshold tau is a hyperparameter whose robustness is ablated in Sec. 6.3, not a quantity fitted to force the reported results. There is no load-bearing self-citation: the authors cite standard external works (HER, UVFA, FPS, PRM-RL) and do not rely on a uniqueness theorem or prior result of their own to justify the method. The Section 7 limitation that the approach 'will also suffer from the accumulated error' when a reliable local policy cannot be learned is a limitation statement, not evidence of circularity; it correctly identifies the assumption of local accuracy on which any compositional planner depends, but the paper does not define success in terms of that assumption. Overall, the central claim is self-contained and empirically checked, so the correct finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method does not introduce new physical entities; the landmark graph is a computational structure. The central assumptions are the low-dimensional manifold structure, local reliability of the learned value function, and the reward-to-distance equivalence. Key hand-chosen numbers are the edge threshold, landmark count, success radius, HER relabeling window, and discount factor.

free parameters (5)

Edge clipping threshold tau
Connects only landmark pairs with estimated distance below tau; must be large enough for connectivity and small enough for local accuracy. Ablation in Fig. 6a shows sensitivity, but the value per environment is not reported.
Number of landmarks selected by FPS
Controls map resolution and planning cost; ablation in Fig. 6a shows the map can become disconnected with too few landmarks. Exact count per environment is not reported.
Goal-reaching distance threshold delta
Defines success in the reward function R(s,a,g) in Section 5.2. Section 7 explicitly admits this threshold needs hand-tuning.
HER relabeling horizon
Section 5.2 replaces HER's goal sampling with near-future goals within a fixed number of steps. This window is not specified and directly influences the local accuracy of the UVFA.
Discount factor gamma
Equation 1 assumes gamma close to 1 via a first-order Taylor expansion, but no experimental gamma value is reported, so the approximation error in converting values to distances is unquantified.

assumptions (5)

domain assumption The state space is a low-dimensional manifold in the ambient space, so a landmark graph can coarsen it without losing connectivity.
Stated at the start of Section 4 ('The state space of our UDMP is a low-dimension manifold in the ambient space') and in Section 7, where high-intrinsic-dimension environments are acknowledged as a limitation.
domain assumption A UVFA trained with HER is locally accurate for nearby state-goal pairs, and this accuracy can be trusted for edge weights below a threshold.
Section 5.3: 'Since UVFA is accurate locally but unreliable for long-term future, we choose to only connect nearby landmarks.' The conclusion admits that weak local policies cause accumulated error.
domain assumption Reward is exactly -1 per step except 0 at the goal, so cumulative return is negative path length and value iteration reduces to shortest-path computation.
This reward structure is assumed in Section 4 and defined in Section 5.2; the equivalence V approximately -d relies on a discount factor near 1 and bounded reward.
standard math Approximating the discounted sum by its first-order Taylor expansion is valid for gamma close to 1.
Section 4, Eq. 1 derivation; the approximation error is bounded by T^2 R_max epsilon, but the actual gamma in experiments is not reported, so the error magnitude is unknown.
domain assumption Farthest point sampling from the replay buffer selects states that cover and expand the visited state space, improving exploration.
Section 5.3 and the ablation in Fig. 6b support this empirically, but no theoretical proof is given for why FPS on UVFA distances expands coverage.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Mapping State Space using Landmarks for Universal Goal Reaching." pith.science (2026). https://pith.science/paper/TA2I6QQH

@misc{pith2026190805451,
  author       = {Pith},
  title        = {Pith review of: Mapping State Space using Landmarks for Universal Goal Reaching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TA2I6QQH}},
  note         = {Machine review of arXiv:1908.05451}
}

read the original abstract

An agent that has well understood the environment should be able to apply its skills for any given goals, leading to the fundamental problem of learning the Universal Value Function Approximator (UVFA). A UVFA learns to predict the cumulative rewards between all state-goal pairs. However, empirically, the value function for long-range goals is always hard to estimate and may consequently result in failed policy. This has presented challenges to the learning process and the capability of neural networks. We propose a method to address this issue in large MDPs with sparse rewards, in which exploration and routing across remote states are both extremely challenging. Our method explicitly models the environment in a hierarchical manner, with a high-level dynamic landmark-based map abstracting the visited state space, and a low-level value network to derive precise local decisions. We use farthest point sampling to select landmark states from past experience, which has improved exploration compared with simple uniform sampling. Experimentally we showed that our method enables the agent to reach long-range goals at the early training stage, and achieve better performance than standard RL algorithms for a number of challenging tasks.

Figures

Figures reproduced from arXiv: 1908.05451 by the authors.

**Figure 2.** The results on FourRoom Environment. Figure 2a shows the sampled landmarks and the [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** The environments we use for continuous control experiments. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Experiments on the continuous control environments. The red curve indicates the perfor [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: AntMaze of multi-level difficulty. Figure 5b and Figure 5c is the average steps and success [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 6.** Figure 6: Figure 6a shows the relationship with the landmarks and clip range in the planner. Figure 6b [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 22 canonical work pages

[1]

Efﬁcient memory-based learning for robot control

Andrew William Moore. Efﬁcient memory-based learning for robot control. Technical report, 1990

work page 1990
[2]

Hierarchical Reinforcement Learning with Hindsight

Andrew Levy, Robert Platt, and Kate Saenko. Hierarchical reinforcement learning with hindsight. arXiv preprint arXiv:1805.08180, 2018

work page Pith review arXiv 2018
[3]

Universal value function approxi- mators

Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxi- mators. In International conference on machine learning, pages 1312–1320, 2015

work page 2015
[4]

Hindsight experience replay

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pages 5048–5058, 2017

2017
[5]

Francesco Bullo and Andrew D. Lewis. Geometric Control of Mechanical Systems, volume 49 of Texts in Applied Mathematics. Springer Verlag, New York-Heidelberg-Berlin, 2004

work page 2004
[6]

Sparse multidimensional scaling using landmark points

Vin De Silva and Joshua B Tenenbaum. Sparse multidimensional scaling using landmark points. Technical report, 2004

work page 2004
[7]

Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction

Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Au- tonomous Agents and Multiagent Systems-Volume 2, pages 761–768. International Foundation for...

work page 2011
[8]

Universal agent for disentangling environments and tasks

Jiayuan Mao, Honghua Dong, and Joseph J Lim. Universal agent for disentangling environments and tasks. 2018

work page 2018

Show all 37 references

[9]

Temporal difference models: Model-free deep RL for model-based control

Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model-free deep RL for model-based control. CoRR, abs/1802.09081, 2018

2018 arXiv
[10]

Continuous control with deep reinforcement learning

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

2015 arXiv
[11]

Learning latent dynamics for planning from pixels

Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551, 2018

2018 arXiv
[12]

Value prediction network

Junhyuk Oh, Satinder Singh, and Honglak Lee. Value prediction network. In NIPS, 2017

2017
[13]

David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Made...

2016
[14]

Model-based planning with discrete and continuous actions

Mikael Henaff, William F Whitney, and Yann LeCun. Model-based planning with discrete and continuous actions. arXiv preprint arXiv:1705.07177, 2017

2017 arXiv
[15]

Universal planning networks

Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal planning networks. CoRR, abs/1804.00645, 2018

2018 arXiv
[16]

Unsupervised visuomotor control through distributional planning networks

Tianhe Yu, Gleb Shevchuk, Dorsa Sadigh, and Chelsea Finn. Unsupervised visuomotor control through distributional planning networks. CoRR, abs/1902.05542, 2019

1902 arXiv
[17]

Modeling the long term future in model-based reinforcement learning

Nan Rosemary Ke, Amanpreet Singh, Ahmed Touati, Anirudh Goyal, Yoshua Bengio, Devi Parikh, and Dhruv Batra. Modeling the long term future in model-based reinforcement learning. 2018

2018
[18]

Value iteration networks

Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value iteration networks. In Advances in Neural Information Processing Systems, pages 2154–2162, 2016

2016
[19]

Gradient-based learning applied to document recognition

Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998

1998
[20]

A formal basis for the heuristic determination of minimum cost paths

Peter E Hart, Nils J Nilsson, and Bertram Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics, 4(2):100–107, 1968. 11

1968
[21]

Rapidly-exploring random trees: A new tool for path planning

Steven M LaValle. Rapidly-exploring random trees: A new tool for path planning. 1998

1998
[22]

Probabilistic roadmaps for path planning in high-dimensional conﬁguration spaces, volume 1994

Lydia Kavraki, Petr Svestka, and Mark H Overmars. Probabilistic roadmaps for path planning in high-dimensional conﬁguration spaces, volume 1994. Unknown Publisher, 1994

1994
[23]

Robot motion planning in learned latent spaces

Brian Ichter and Marco Pavone. Robot motion planning in learned latent spaces. CoRR, abs/1807.10366, 2018

2018 arXiv
[24]

Qureshi and Michael C

Ahmed H. Qureshi and Michael C. Yip. Deeply informed neural sampling for robot motion planning. CoRR, abs/1809.10252, 2018

2018 arXiv
[25]

Towards learning abstract representations for locomotion planning in high-dimensional state spaces

Tobias Klamt and Sven Behnke. Towards learning abstract representations for locomotion planning in high-dimensional state spaces. arXiv preprint arXiv:1903.02308, 2019

1903 arXiv
[26]

Prm-rl: Long-range robotic navigation tasks by combining reinforcement learning and sampling-based planning

Aleksandra Faust, Kenneth Oslund, Oscar Ramirez, Anthony Francis, Lydia Tapia, Marek Fiser, and James Davidson. Prm-rl: Long-range robotic navigation tasks by combining reinforcement learning and sampling-based planning. In 2018 IEEE International Conference on Robotics and Au...

2018
[27]

Semi-parametric topological memory for navigation

Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. arXiv preprint arXiv:1803.00653, 2018

2018 arXiv
[28]

Composable planning with attributes

Amy Zhang, Adam Lerer, Sainbayar Sukhbaatar, Rob Fergus, and Arthur Szlam. Composable planning with attributes. arXiv preprint arXiv:1803.00512, 2018

2018 arXiv
[29]

Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation

Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems, pages 3675–3683, 2016

2016
[30]

Data-efﬁcient hierarchical reinforcement learning

Oﬁr Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efﬁcient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, pages 3303– 3313, 2018

2018
[31]

Benchmarking deep reinforcement learning for continuous control

Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. CoRR, abs/1604.06778, 2016

2016 arXiv
[32]

Multi-goal reinforcement learning: Challenging robotics environments and request for research

Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojciech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for resear...

2018 arXiv
[33]

Computing the shortest path: A search meets graph theory

Andrew V Goldberg and Chris Harrelson. Computing the shortest path: A search meets graph theory. In Proceedings of the sixteenth annual ACM-SIAM symposium on Discrete algorithms, pages 156–165. Society for Industrial and Applied Mathematics, 2005

2005
[34]

k-means++: The advantages of careful seeding

David Arthur and Sergei Vassilvitskii. k-means++: The advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, pages 1027–1035. Society for Industrial and Applied Mathematics, 2007

2007
[35]

Riedmiller

V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013

2013 arXiv
[36]

Mujoco: A physics engine for model-based control

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. IEEE, 2012

2012
[37]

Openai gym, 2016

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016. 12

2016

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Efﬁcient memory-based learning for robot control

Andrew William Moore. Efﬁcient memory-based learning for robot control. Technical report, 1990

work page 1990

[2] [2]

Hierarchical Reinforcement Learning with Hindsight

Andrew Levy, Robert Platt, and Kate Saenko. Hierarchical reinforcement learning with hindsight. arXiv preprint arXiv:1805.08180, 2018

work page Pith review arXiv 2018

[3] [3]

Universal value function approxi- mators

Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxi- mators. In International conference on machine learning, pages 1312–1320, 2015

work page 2015

[4] [4]

Hindsight experience replay

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pages 5048–5058, 2017

2017

[5] [5]

Francesco Bullo and Andrew D. Lewis. Geometric Control of Mechanical Systems, volume 49 of Texts in Applied Mathematics. Springer Verlag, New York-Heidelberg-Berlin, 2004

work page 2004

[6] [6]

Sparse multidimensional scaling using landmark points

Vin De Silva and Joshua B Tenenbaum. Sparse multidimensional scaling using landmark points. Technical report, 2004

work page 2004

[7] [7]

Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction

Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Au- tonomous Agents and Multiagent Systems-Volume 2, pages 761–768. International Foundation for...

work page 2011

[8] [8]

Universal agent for disentangling environments and tasks

Jiayuan Mao, Honghua Dong, and Joseph J Lim. Universal agent for disentangling environments and tasks. 2018

work page 2018