{"id":"d56d6434-9735-4a64-a0c7-c789914b4a28","arxiv_id":"1908.05451","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An agent that plans on a landmark graph built from a locally accurate value function reaches long-range goals faster and more reliably than model-free baselines in sparse-reward environments.","lead":"This paper builds a map of the states an agent has already visited, using sampled landmarks and a value network that is accurate over short distances, and plans routes toward distant goals on that map. The result is a goal-conditioned reinforcement learning method that reaches far-away goals earlier and more reliably than standard baselines in maze and robotic control tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Map quality hinges on an unvalidated claim that UVFA distances below clip_bound are reliable, and the threshold is certified by the same potentially biased estimator.","rationale":"The paper's core mechanism is a graph whose nodes are landmarks and whose edges are UVFA-based distance estimates, used with shortest-path planning to obtain long-range value estimates. The entire construction relies on the premise that edge weights are reliable whenever the UVFA output is below the clip threshold. That premise is not established: the threshold is chosen using the same estimator whose reliability it is supposed to certify, and the reported experiments do not isolate edge-weight accuracy. This is a genuine correctness risk, but it is not a demonstrated contradiction or a fatal flaw; the method could still work if the local estimates are in fact reliable within the threshold, and the ablation results provide indirect evidence that some reasonable threshold choices work. The reader's conditional verdict is therefore appropriate: the central claim is plausible and directionally supported, but the key assumption needs explicit validation before the claim can be accepted as established. My concern sharpens the reader's weakest assumption by pointing to the circularity in using Q to define reliability, and by proposing a concrete diagnostic that would settle whether this assumption holds in practice.","tokens_in":11682,"tokens_out":3386,"duration_ms":36073,"concrete_test":"On FourRoom or AntMaze with known ground-truth distances, take a fixed checkpoint (e.g., 0.8M steps in AntMaze). (1) For every landmark pair with w_ij <= clip_bound, compare w_ij with the true shortest-path distance d*; report median relative error and the fraction of connected pairs whose true distance exceeds clip_bound. (2) Compute the recall of truly nearby pairs (d* <= clip_bound) that are connected. (3) Re-run the planner twice: once with learned edge weights and once with ground-truth weights on the identical connected edge set, and compare success rates. If learned-edge success is substantially lower, local UVFA bias is the bottleneck; if the gap is small, the map is robust to edge-weight error.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 (lines 4-8) adds an edge between landmarks v_i and v_j only when w_ij = min_a -Q(v_i,v_j,a) <= clip_bound, then runs Bellman-Ford over these edges. The same learned Q thus decides which landmark pairs are 'nearby' and supplies the edge cost. If Q underestimates the distance for a genuinely long pair, that pair is inserted as a short edge and creates a false shortcut; if Q overestimates, the graph fragments. The paper never measures whether the threshold actually selects reliable local estimates: the FourRoom MDE in Fig. 2b is averaged over all state-goal pairs, not conditioned on connected landmark edges, and no analogous calibration is reported for the continuous environments. Section 6.3's ablation reports only end success rates for a few tau values. Since the central claim is that shortest paths over locally accurate edges yield accurate long-range values, this unquantified reliance on Q's own estimate to certify its reliability is the load-bearing assumption. The authors' Section 7 concession ('this approach will also suffer from the accumulated error') confirms the dependency rather than bounding it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper addresses universal goal reaching in sparse-reward MDPs whose state space is a low-dimensional manifold. The proposed method, 'Planning with State-space Mapping' (Algorithm 1), trains a goal-conditioned Q-function with HER, samples landmark states from the replay buffer using farthest-point sampling, connects landmark pairs whose learned Q-distance falls below a clip threshold to form a directed weighted graph, runs Bellman-Ford to compute pairwise landmark distances, and at execution time selects the next landmark that minimizes the sum of the local value from the current state to a landmark and the planned distance from that landmark to the goal. The paper reports lower mean distortion error than HER on a FourRoom grid with ground-truth shortest paths, higher success rates on 2DReach, 2DPush, PointMaze, AntMaze, Complex AntMaze, Acrobot, and BlockedFetchReach/FetchPush, and an advantage over HIRO on a large AntMaze in a table. Ablations study the clip threshold, the number of landmarks, FPS vs uniform sampling, and local accuracy of the learned value function.","tokens_in":11874,"tokens_out":5106,"duration_ms":47017,"significance":"If the empirical claims hold, the paper makes a useful conceptual contribution: it decomposes long-horizon value estimation into a graph-based global planner over landmark nodes and a locally trained UVFA, thereby avoiding extrapolation over long distances. The FourRoom experiment (Fig. 2) is a genuine validation because mean distortion error is measured against ground-truth shortest paths. The method is simple and the Algorithm 1 pseudocode is clear. However, the evidence is not yet at the level needed to support the central claim: the key reliability assumption about local edge weights is not directly tested, and the continuous-control comparisons lack seeds, error bars, and full hyperparameter reporting. The paper is therefore promising but requires additional experiments and diagnostics before publication.","major_comments":[{"comment":"The graph's edge set is determined by the same learned Q that supplies the edge weights: an edge (v_i, v_j) is included iff min_a -Q(v_i, v_j, a) <= clip_bound. The manuscript asserts that 'UVFA is accurate locally but unreliable for long-term future' (Sec. 5.3) but never measures accuracy conditioned on the pairs that are actually connected in the graph. Fig. 2b's mean distortion error is averaged over all state-goal pairs, not over landmark pairs that pass the threshold; no analogous calibration is reported for the continuous environments. Section 6.3 reports only end success rates for a few tau values. This is load-bearing because if Q underestimates a genuinely long-range pair, that pair enters the graph as a false shortcut, and if it overestimates, the graph fragments. Please add a diagnostic (e.g., in FourRoom, compare UVFA edge weights with ground-truth distances for threshold-passing landmark pairs; in continuous environments, report edge-connectivity statistics and the sensitivity of success to tau) and condition the MDE on graph edges.","section":"Section 5.3 / Algorithm 1, lines 4-8"},{"comment":"The conversion V ≈ -d rests on the bound T^2 Rmax(1-gamma) -> 0. The paper does not report gamma, Rmax, or T for the experiments. For AntMaze with horizon 500 and gamma=0.99, T^2 Rmax(1-gamma) = 2500 Rmax, which is not close to zero. This does not necessarily invalidate the method, but it means the 'distances' used as edge weights and planned over are not justified quantitatively by the stated derivation. Please either state the actual gamma and horizon values, replace Eq. (1) with an episodic undiscounted formulation, or provide a bound that is meaningful in the experimental regime.","section":"Section 4, Eq. (1) and Algorithm 1"},{"comment":"Figure 4 and Table 1 report success-rate curves and values without error bars or the number of seeds. Table 1 is introduced as a comparison with HIRO and HAC, but HAC rows do not appear, and the text says HAC 'couldn't make it work well' without giving numbers. Hyperparameters such as the number of landmarks, clip_bound/tau, goal-reaching delta, HER relabeling horizon, network architecture, learning rate, and replay buffer size are not stated. Without these, the central empirical claim that the planner 'achieve better performance than standard RL algorithms' (Abstract) cannot be independently evaluated. Please add multi-seed mean/std, a full hyperparameter table, and the missing HAC results, or remove HAC from the comparison.","section":"Section 6 / Table 1"}],"minor_comments":[{"comment":"Algorithm 1 is referred to as 'Algorithm 10' in the text; correct the reference.","section":"Section 5.3"},{"comment":"The text says '2DReach and 2DPush ... shown in Figure 4b', but 2DReach appears in Figure 4a.","section":"Section 6.2.2"},{"comment":"The legend contains 'uni00A0' artifacts; clean up the labels.","section":"Figure 2b"},{"comment":"The caption says the red curve indicates the performance of our method, but the curves in the figure are not visually distinguished in grayscale; use distinct markers or colors and a legend.","section":"Figure 4"},{"comment":"The notation 'T2Rmax' should be 'T^2 Rmax'; also clarify the role of epsilon in the Taylor expansion.","section":"Section 4"},{"comment":"The modification of HER 'ensuring that the replaced goals are sampled from the near future within a fixed number of steps' is never quantified; state the fixed number.","section":"Section 5.2"},{"comment":"In the 'Ours Sparse' row, the success rate reaches 0.5 at 0.5M steps and remains 0.5 at 1.5M and 2M; clarify whether this is a plateau or a reporting artifact.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the core idea is interesting and the FourRoom experiment provides a useful sanity check, but the experimental reporting is not yet complete enough for the central claims to be verified. In particular, the missing HAC rows in Table 1 and the absence of seeds/error bars in Figure 4 should be addressed before the paper can be considered for acceptance. The paper would also benefit from a clear statement of which code and hyperparameters will be released."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know upfront. First, the core idea is genuinely useful: instead of forcing a single UVFA to memorize long-range values, build a dynamic graph from replay states chosen by farthest-point sampling, connect only nearby landmarks with learned local distance estimates, and run shortest-path search. That is a clean way to factor long-horizon goal reaching. Second, the paper is honest about its own limits, but the empirics don't yet back the central claim about local reliability.\n\nWhat's new is the assembly: FPS plus UVFA-weighted edges plus Bellman-Ford. Individually these are known, but the dynamic construction is a reasonable contribution. The FourRoom experiment is a real strength — validating against ground-truth shortest paths rather than just reporting success rates is the right kind of check, and the early MDE improvement is credible. The related work is also careful: the paper correctly distinguishes itself from PRM-RL (which needs a physical engine) and from Savinov et al. (which needs a supervised connectivity classifier). The FPS-vs-uniform ablation shows a concrete exploration benefit.\n\nThe weak spots are real but not fatal. The curves are single-run with no error bars, and hyperparameters like the clip threshold, landmark count, and goal-distance delta are only partially reported. I would like to see comparisons to PRM-RL and Savinov, since those are the closest graph-planning baselines and the paper discusses them without testing. The deeper issue is the threshold tau. The same learned Q supplies both the edge weights and the criterion for whether a pair of landmarks is close enough to trust. If Q underestimates distance, you get false shortcuts; if it overestimates, the graph fragments. The FourRoom MDE in Fig. 2b is averaged over all pairs, not conditioned on connected edges, so it doesn't tell us whether the threshold actually selects reliable local estimates. Section 7's concession about accumulated error confirms the dependency without bounding it. This is a placeholder for future work, not a contradiction of the method, but it should be quantified for the central claim to land.\n\nOverall, the paper is for researchers in goal-conditioned RL and model-based planning who want a simple sample-based way to extend locally accurate value functions to long horizons. It is worth a serious referee. A good reviewer should ask for multi-seed runs, a hyperparameter table, and some diagnostic about edge reliability under varying tau. My recommendation: send to peer review, but expect revision.","headline":"A plausible landmark-graph planner for goal-conditioned RL that gets the core idea right, but the evaluation doesn't yet quantify the load-bearing assumption that local UVFA estimates are reliable within the clipping threshold.","tokens_in":12410,"tokens_out":1967,"would_cite":false,"duration_ms":20888,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A landmark map turns local value estimates into long-range goal reaching in RL.","keywords":["universal value function approximation","goal-conditioned reinforcement learning","landmark-based map","farthest point sampling","sparse rewards","shortest path planning","hierarchical RL","long-range goals"],"falsifier":"In a maze with known ground-truth distances, compute shortest-path distances over the landmark graph using locally verified UVFA edge weights and compare them to ground-truth distances for all goal pairs; if the planned length is systematically wrong even when every edge's local estimate is correct, the central claim fails.","tokens_in":11462,"feed_emoji":"🗺️","tokens_out":3447,"duration_ms":32860,"temperature":0.7,"pith_summary":"This paper argues that for goal-reaching tasks in large, sparse-reward environments, a single learned value network cannot reliably estimate distances to faraway goals. Instead, the agent should maintain a small graph map of the visited state space: nodes are landmark states sampled by farthest-point sampling, and edges connect nearby landmarks with distances estimated by a locally accurate value function. A shortest-path search over this map propagates trustable local estimates across long horizons, so the agent can reach distant goals early in training and outperform standard goal-conditioned reinforcement learning. If correct, long-range value estimation can be factored into local learning plus global graph search, reducing sample complexity and dependence on network extrapolation.","feed_headline":"A landmark map lets RL agents reach distant goals early in training","feed_subtitle":"Long-range values come from graph search over trusted local estimates, not from one overstretched network.","key_machinery":"The central object is the landmark-based map: a directed weighted graph whose nodes are states selected from the replay buffer via farthest-point sampling (FPS), and whose edges are UVFA distance estimates between nearby landmarks. FPS iteratively picks the state farthest from already chosen landmarks, spreading landmarks toward the boundary of the visited space and improving exploration. The map does the work of global value propagation: the local UVFA supplies trustworthy edge weights and local control, while the graph supplies topology and long-range routing.","core_discovery":"The central claim is that the all-pairs value estimation problem in universal goal reaching can be coarsened into a landmark graph without losing routing accuracy, provided the universal value function approximator is accurate only in a local neighborhood. The paper builds a directed weighted graph from replay-buffer states; it connects landmark pairs only when the UVFA-estimated distance is below a threshold $\\tau$, then runs Bellman-Ford to obtain pairwise landmark distances. The next subgoal is chosen by minimizing $d(s, \\ell_i) + d(\\ell_i, g)$, so the long-range distance estimate is a piecewise shortest path over trusted local estimates rather than a single network extrapolation. Empirically, this planner gives lower mean distortion error on value estimates and higher success rates on maze and manipulation tasks, especially in the early training stage.","pith_inferences":["The same coarsening idea could apply to any learned local distance or reachability signal, not just UVFA, provided edge reliability can be certified.","A testable extension is to grow the edge threshold $\\tau$ online as the local value estimates become more accurate, potentially removing the hand-tuned clipping parameter.","If landmark edge weights are consistently biased, shortest-path distances may still be accurate only up to an additive bias; evaluating planned lengths against ground-truth shortest paths, rather than only success rates, would expose this."],"forward_implications":["Long-horizon goals in sparse-reward environments become reachable early in training because planning decouples local control from global routing.","A goal-conditioned value network only needs local accuracy, reducing the sample complexity and network capacity required compared with a global UVFA.","Exploration improves because FPS landmarks tend to lie at the frontier of experienced states, guiding the agent outward.","The planner combines with HER-trained DQN and DDPG and yields stable success on long-horizon mazes where standard HER fails.","The graph-based distance estimator naturally extends to state-goal pairs never experienced together, as long as a path through landmarks exists."],"supporting_citations":[{"why":"Defines the UVFA objective and the universal value function that the local network approximates.","marker":"[3]"},{"why":"Provides hindsight experience replay, the relabeling scheme used to train the local UVFA in sparse-reward settings.","marker":"[4]"},{"why":"Supplies the farthest-point sampling / k-means++ seeding strategy used to select landmarks from the replay buffer.","marker":"[34]"},{"why":"Serves as the hierarchical RL baseline compared on AntMaze in both sparse and dense reward settings.","marker":"[30]"},{"why":"Serves as the hierarchical RL baseline (HAC) discussed in the large AntMaze comparison.","marker":"[2]"},{"why":"Contrasts value iteration networks, which require a predefined map, against the constructed landmark map.","marker":"[18]"},{"why":"Contrasts PRM-RL, a prior shortest-path-plus-RL planner that depends on a physical engine for edge connections.","marker":"[26]"}],"fun_headline_variants":["Landmark graph turns distant goals into short hops for RL agents","Farthest-point landmarks make long-range RL goals achievable early","Hierarchical landmark map lets RL agents plan to far goals fast","Landmark subgoals unlock early success on long-horizon RL tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The graph is only as trustworthy as the local UVFA distance estimates; if those estimates are biased within the threshold $\\tau$, every edge inherits the bias and the planned shortest path is wrong.","fun_headline_variants_meta":{"raw":{"variants":["Landmark graph turns distant goals into short hops for RL agents","Farthest-point landmarks make long-range RL goals achievable early","Hierarchical landmark map lets RL agents plan to far goals fast","Landmark subgoals unlock early success on long-horizon RL tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00062,"raw_usage":{"total_tokens":2851,"prompt_tokens":895,"completion_tokens":1956,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":1882}},"tokens_in":511,"tokens_out":1956,"duration_ms":12039,"temperature":1.0,"reasoning_tokens":1882,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:13:36.296238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a maze with known ground-truth distances, compute shortest-path distances over the landmark graph using locally verified UVFA edge weights and compare them to ground-truth distances for all goal pairs; if the planned length is systematically wrong even when every edge's local estimate is correct, the central claim fails.","supporting_citations":[{"cited_title":"Universal value function approxi- mators","cited_arxiv_id":null,"evidence_quote":"Defines the UVFA objective and the universal value function that the local network approximates."},{"cited_title":"Data-efficient hierarchical reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Serves as the hierarchical RL baseline compared on AntMaze in both sparse and dense reward settings."},{"cited_title":"Hierarchical Reinforcement Learning with Hindsight","cited_arxiv_id":"1805.08180","evidence_quote":"Serves as the hierarchical RL baseline (HAC) discussed in the large AntMaze comparison."},{"cited_title":"Value iteration networks","cited_arxiv_id":null,"evidence_quote":"Contrasts value iteration networks, which require a predefined map, against the constructed landmark map."},{"cited_title":"Prm-rl: Long-range robotic navigation tasks by combining reinforcement learning and sampling-based planning","cited_arxiv_id":null,"evidence_quote":"Contrasts PRM-RL, a prior shortest-path-plus-RL planner that depends on a physical engine for edge connections."}],"review_version":1}