{"id":"7293159f-7dde-43f8-8ea8-13d39d570fcf","arxiv_id":"2507.12977","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A reinforcement learning fine-tuning method with dynamic reward thresholding lets diffusion motion planners directly optimize non-differentiable safety and goal-reaching metrics, improving collision rate and success rate on CrowdNav and ETH-UCY.","lead":"This paper trains diffusion-based robot planners with reinforcement learning, using collision rate and goal success as rewards instead of differentiable loss functions. A dynamic thresholding trick keeps the reward signal informative when collisions or successes are rare, and the method beats several baselines on pedestrian navigation datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dynamic thresholding can decouple training reward from the fixed evaluation thresholds; the reported gains may depend on the adaptation path.","rationale":"The reader's CONDITIONAL verdict is appropriate. Our stress-test identifies a deeper version of the reader's concern: not just missing hyperparameters, but a potential misalignment between the dynamically shaped reward and the fixed evaluation metric. The paper's own Fig. 5 shows threshold choice strongly affects results, yet the dynamic threshold path is not reported. A concrete test would settle whether the headline metrics are robust. Since the concern can be addressed by additional experiments and does not demonstrate a fatal flaw in the method's logic, the verdict remains conditional. The reader's weakest_assumption focused on hyperparameter specification; I partially agree, but the more precise issue is the unanchored threshold and the lack of evidence that the adaptive reward converges to the evaluation metric.","tokens_in":11860,"tokens_out":6268,"duration_ms":73577,"concrete_test":"Re-run CrowdNav training with dynamic thresholding disabled but with the fixed evaluation thresholds as rewards (collision 0.6m, success 0.2m); also run with fixed collision thresholds 0.3m and 0.9m. If the fixed-evaluation-threshold run achieves collision rate <= 0.02 and success rate >= 0.48 (comparable to Table I), then dynamic thresholding is not necessary for the reported gains. Additionally, log the final epsilon values from the dynamic run: if the final collision threshold is > 0.6m (looser than evaluation), the training reward is not aligned with the metric, and the improvement mechanism is unclear.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central contribution is the dynamic thresholding algorithm (Algorithm 1) that adjusts the collision/success threshold so the batch-mean binary reward stays near B/2. The evaluation, however, uses fixed thresholds (CrowdNav: 0.6m collision, 0.2m success). Starting from these thresholds, the algorithm can drift: as the model improves, the collision threshold tightens to keep half the batch 'colliding', which is beneficial; but if the model degrades or the adaptation rate is too aggressive, the threshold can loosen, making the training reward easier while the fixed-threshold metric does not improve. The paper never reports the final thresholds after training, the threshold trajectory, or a sensitivity analysis over epsilon_init, alpha, J, or delta. Table IV's DT ablation covers only one reward configuration and reports no variance or multiple seeds. Thus the headline numbers (collision 0.0150, success 0.4931) could be tied to a specific adaptation path rather than a robust property of the planner. This is load-bearing because the paper's novelty claim rests on dynamic thresholding, not on vanilla DDPO; if the threshold mechanism is misaligned with the evaluation metric, the claimed state-of-the-art performance is not explained.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a reinforcement learning-based fine-tuning scheme for diffusion-based motion planners, built on DDPO, to optimize non-differentiable rewards derived from collision rate, success rate, and discomfort rate. To handle sparse rewards, it introduces a dynamic thresholding procedure (Algorithm 1) that adjusts the threshold used in each binary reward so that the batch mean reward stays near B/2. Experiments on CrowdNav and ETH-UCY report lower collision rate, higher success rate, and lower ADE compared to DTPP, DIPP, GameTheoretic, the backbone, and gradient-guidance baselines. Ablations compare single- versus multi-reward training, differentiable versus non-differentiable rewards, and training with versus without dynamic thresholding.","tokens_in":11997,"tokens_out":9512,"duration_ms":109352,"significance":"If the empirical claims hold, the paper makes a useful contribution: it demonstrates a practical way to directly optimize non-differentiable planning metrics in diffusion planners and shows consistent improvements on safety and effectiveness metrics over surrogate differentiable losses. The method is grounded in the standard DDPO gradient (Eq. 7), and the multi-reward formulation is flexible. The authors also provide a direct comparison against gradient guidance and an ablation isolating dynamic thresholding, which are informative. However, the strength of the evidence is limited by missing uncertainty estimates, under-specified hyperparameters of the key new algorithm, and an incomplete description of the baseline comparison protocol, so the significance is conditional on additional validation.","major_comments":[{"comment":"Algorithm 1 introduces the paper's central dynamic thresholding mechanism, but none of its controlling hyperparameters are specified: the initial threshold epsilon_init for each reward (the fixed evaluation thresholds in Section IV-B are not explicitly identified with epsilon_init), the deviation delta, the adaptation rate schedule alpha(j), and the maximum iterations J are all absent, and no threshold trajectories or final learned thresholds are reported. Because the algorithm deliberately adjusts epsilon away from the evaluation thresholds as the policy changes, the reported improvements (e.g., collision rate 0.0150 in Table I) may depend on the specific adaptation path rather than on a robust property of the learned planner. A sensitivity analysis over these hyperparameters, at least one training curve of epsilon, and the exact values used for the reported runs are necessary to support the central claim.","section":"Algorithm 1, Sections III-E and IV-D"},{"comment":"All quantitative results in Tables I-IV are single point estimates: no standard deviations, confidence intervals, or numbers of seeds are reported. This is particularly problematic for the small margins (e.g., ADE 0.1984 vs 0.1991 in Table I, success rate 0.4931 vs 0.4987 in Table II) and for the small collision-rate values, where stochasticity can change rankings. The dynamic-thresholding ablation in Table IV is also a single configuration. The state-of-the-art claim requires mean and variance over at least several independent runs, ideally with paired comparisons.","section":"Tables I-IV"},{"comment":"Section V-B claims that non-differentiable rewards generally outperform differentiable rewards, but Table IV shows the opposite when dynamic thresholding is not used: collision cost achieves 0.0216 collision rate without DT, while collision rate achieves 0.0232. The superiority of the non-differentiable reward appears only in the w/ DT row. This interaction between reward type and dynamic thresholding should be acknowledged, because it complicates the interpretation of Table III and the paper-level claim that non-differentiable rewards are the source of the gains.","section":"Section V-B and Table IV"},{"comment":"The comparison protocol for the main baselines is under-specified. Section IV-C states that all baseline models are conventionally trained with gradients from L2 with GT trajectory and collision cost, but it does not say whether DTPP, DIPP, and GameTheoretic use their original architectures, whether they are retrained under the same frozen-forecaster condition, or whether the numbers come from prior papers. Without a clear description of backbone, training schedule, and evaluation protocol, the state-of-the-art claim in Table I cannot be independently assessed.","section":"Section IV-C and Section IV-D"}],"minor_comments":[{"comment":"The objective sums R(y_k, F) over all diffusion steps, but the reward is defined only for the final trajectory and Eqs. (6)-(7) use R(hat y, F). This is inconsistent; either remove the sum and use R(y_0) or define intermediate rewards.","section":"Section III-C, Eq. (5)"},{"comment":"The sentence giving thresholds, '0.6 m, 0.2 m for CrowdNav and 0.2 m, 0.5 m for ETH-UCY', does not state which threshold corresponds to collision rate and which to success rate; please label them explicitly.","section":"Section IV-B"},{"comment":"The discomfort rate is used as a reward and metric, but its definition (the jerk threshold and how jerk is computed) is never given; this should be added to Section IV-B.","section":"Section V-A and Tables II-III"},{"comment":"The reward-based training details omit standard RL and optimization hyperparameters such as learning rate, number of fine-tuning epochs, number of diffusion steps K, and the alpha(j) schedule used for dynamic thresholding; including these is needed for reproducibility.","section":"Section IV-D"},{"comment":"The caption misspells CrowdNav as 'CrowNav'; please correct it.","section":"Table III caption"},{"comment":"The paper does not mention code or model release; given the number of unspecified training details, a public implementation would substantially help reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The work is within scope for the journal, and the core idea is reasonable. The main risk is reproducibility: the dynamic-thresholding algorithm is the paper's novelty, but its hyperparameters, threshold trajectories, and sensitivity are not reported, and all quantitative claims lack variance estimates. I recommend requesting these additions, as well as a clearer description of the baseline comparison protocol, before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimate application of DDPO to diffusion motion planning, with a dynamic thresholding twist that seems to help. The CrowdNav numbers look good, but the paper undersells its own uncertainty: no seeds, no variance, and the dynamic thresholding hyperparameters are never specified. I would send it to review, but the referee should push hard on the threshold-adaptation story.\n\nWhat is actually new: DDPO is prior work, and applying it to planning is a natural step. The contribution is Algorithm 1, which adjusts binary reward thresholds so batch-mean reward stays near B/2. That is a sensible response to sparse rewards, and Table IV suggests it helps for both differentiable and non-differentiable rewards. Table III is also instructive: optimizing the discrete metric directly beats optimizing a differentiable surrogate, which is the paper's core claim, and it holds. I also got a good sign from the ADE number: ADE is not in the reward, yet it improves, so the planner is not simply gaming the threshold.\n\nSoft spots. The stress-test worry is real. Training uses adaptive thresholds; evaluation uses fixed thresholds. If the adaptation path drifts, the training reward can become easier while the fixed-threshold metric stagnates. The paper never reports final training thresholds, threshold trajectories, or any sensitivity analysis over epsilon_init, alpha, J, delta. Table IV's DT ablation is a single configuration with no variance. Without that, the headline collision 0.0150 could be tied to a specific adaptation run. That is load-bearing because the novelty claim rests on dynamic thresholding. Also, no code, no seeds, no variance anywhere. For a method paper, that is a significant reproducibility gap.\n\nOne more minor point: the comparison to gradient guidance [15] is a bit thin, with only one differentiable cost tried. Not a flaw, just a limited baseline.\n\nThe math (Eq. 7) is standard DDPO with baseline normalization; no red flags there. The citation pattern seems fine, and DDPO is properly credited.\n\nBottom line: worth a serious referee. The core idea is sensible and the evidence is suggestive but incomplete. I would accept it for review with the expectation of major revisions around reporting and threshold robustness. For my own work, I would hold off citing until the authors release code or at least the missing hyperparameters.","headline":"A legitimate DDPO-for-planning application with a useful dynamic thresholding trick; the headline numbers are plausible but the missing variance and threshold specification make the central claim under-supported.","tokens_in":12609,"tokens_out":2349,"would_cite":false,"duration_ms":26375,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Diffusion motion planners can be trained to directly optimize collision and goal-success metrics by using them as RL rewards, with dynamic thresholds shaping a dense signal.","keywords":["diffusion models","motion planning","reinforcement learning","non-differentiable rewards","collision avoidance","dynamic thresholding","trajectory prediction","autonomous navigation"],"falsifier":"Train the same backbone with the dynamic thresholding algorithm under several initial thresholds and adaptation schedules, then evaluate at the fixed CrowdNav thresholds; if collision and success rates fluctuate with the initial hyperparameters or fall to backbone level when the initial threshold is far from the evaluation threshold, the claimed benefit of dynamic thresholding would be refuted. A simpler version would run the Table IV ablation with a grid of deviation and adaptation-rate values and check whether the 0.0146 collision rate persists.","tokens_in":11561,"feed_emoji":"🤖","tokens_out":4856,"duration_ms":51260,"temperature":0.7,"pith_summary":"This paper argues that diffusion-based motion planners, normally trained by denoising toward ground-truth trajectories, can be fine-tuned with reinforcement learning to directly optimize non-differentiable planning metrics such as collision rate, goal success, and discomfort rate. The authors show that the central obstacle, sparse rewards from threshold-based metrics, can be handled by a dynamic thresholding procedure that keeps the batch mean reward near half its maximum, producing a dense and stable training signal. On CrowdNav and ETH-UCY, the fine-tuned planner reduces CrowdNav collision rate from 0.0628 to 0.0150 and raises success rate from 0.3540 to 0.4931, outperforming differentiable baselines and gradient-guidance alternatives. The claim is that reward-driven training on the deployed metrics themselves, not surrogate losses, is what delivers the gains.","feed_headline":"RL fine-tuning cuts diffusion-planner collisions fourfold","feed_subtitle":"Training on collision and success metrics as rewards beats differentiable objectives on pedestrian benchmarks.","key_machinery":"The mechanism is a multi-step MDP view of reverse diffusion plus a dynamic thresholding reward shaper. Each reverse denoising step from noise to trajectory is a state-action transition whose log-probability is weighted by an advantage computed from the non-differentiable reward of the final trajectory; the threshold parameter in each metric (collision distance, goal distance, jerk) is adapted per iteration so that the batch mean reward approaches B/2, turning a mostly-zero signal into a signal that distinguishes good from bad samples. All gradient information for denoising remains differentiable; the reward itself never needs a gradient.","core_discovery":"On the paper's own terms, the central discovery is that a diffusion planner's denoising process can be treated as sequential decisions and updated by policy gradient with a reward that directly evaluates the generated trajectory by collision rate and success rate. Because these rewards are threshold-based and often sparse, the paper introduces dynamic thresholding: at each iteration, the threshold defining a collision or a success is raised or lowered so that the batch mean reward stays close to B/2, the midpoint of the batch; reward then weights the likelihood of each denoising step through an importance-sampled policy gradient with a baseline normalized by the reward standard deviation. The measured consequence is large gains on both safety and effectiveness metrics relative to the backbone and to methods that use differentiable collision costs, with the non-differentiable discrete rewards outperforming their differentiable counterparts in the ablations.","pith_inferences":["If dynamic thresholding works by keeping the batch reward near B/2, then training thresholds differ from the fixed evaluation thresholds; a natural testable extension is to anneal thresholds from adaptive values to the deployment thresholds during training and measure whether the gains persist.","The sparse-reward argument suggests the same scheme could apply to any threshold-based planning metric (comfort, rule violations, time-to-collision) or to other generative policies, since the method does not rely on domain-specific differentiable structure.","The ablation showing that performance drops as thresholds become less sensitive implies that threshold choice may matter as much as the RL update itself; sweeping initial thresholds and adaptation rates would clarify how much of the gain comes from dynamic adjustment versus the reward definition."],"forward_implications":["A planner fine-tuned this way can be steered toward whichever metric is rewarded; optimizing success rate alone pushes collision rate down as well, indicating that the metrics are coupled.","Dynamic thresholding benefits even differentiable rewards like collision cost, but gives the largest improvement on discrete non-differentiable rewards.","The method can incorporate arbitrary numbers and types of rewards (safety, effectiveness, comfort) weighted by importance factors, so deployment-specific objectives can be added without rearchitecting.","Because the forecaster is kept frozen, the reported gains come from planner training alone, suggesting the planning module can be improved independently of prediction quality.","The final model improves ADE as well, so safety gains do not come at the cost of fidelity to ground-truth trajectories."],"supporting_citations":[{"why":"Supplies the policy-gradient MDP formulation for training diffusion models that the paper adapts.","marker":"[19]"},{"why":"Provides the denoising diffusion probabilistic model objective and backbone used by the planner.","marker":"[20]"},{"why":"Supplies the game-theoretic forecaster, the collision cost baseline, and the CrowdNav and ETH-UCY collision and success thresholds.","marker":"[45]"},{"why":"State-of-the-art differentiable planning baseline compared on both datasets.","marker":"[47]"},{"why":"Differentiable integrated prediction-planning baseline compared on both datasets.","marker":"[48]"},{"why":"Gradient-guidance baseline that optimizes differentiable collision cost at sampling time.","marker":"[15]"},{"why":"CrowdNav dataset and its crowd-robot interaction setting used for training and evaluation.","marker":"[37]"},{"why":"ETH subset of the ETH-UCY benchmark used as the validation scene.","marker":"[43]"},{"why":"UCY subsets of the ETH-UCY benchmark used in training.","marker":"[44]"}],"fun_headline_variants":["Non-differentiable rewards outperform smooth ones for diffusion planners","Policy gradient with discrete rewards boosts diffusion motion planning","Dynamic thresholding trains diffusion planners with sparse rewards","Reinforcement learning fine-tunes diffusion planners on safety","For diffusion planners, non-differentiable rewards are the key"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that keeping the batch mean reward near half the maximum, via the dynamic threshold updates, produces a stable and informative training signal that transfers to the fixed thresholds used at evaluation; the algorithm's hyperparameters and sensitivity are not reported.","fun_headline_variants_meta":{"raw":{"variants":["Non-differentiable rewards outperform smooth ones for diffusion planners","Policy gradient with discrete rewards boosts diffusion motion planning","Dynamic thresholding trains diffusion planners with sparse rewards","Reinforcement learning fine-tunes diffusion planners on safety","For diffusion planners, non-differentiable rewards are the key"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000261,"raw_usage":{"total_tokens":1570,"prompt_tokens":897,"completion_tokens":673,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":595}},"tokens_in":513,"tokens_out":673,"duration_ms":7552,"temperature":1.0,"reasoning_tokens":595,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:33:13.089682+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same backbone with the dynamic thresholding algorithm under several initial thresholds and adaptation schedules, then evaluate at the fixed CrowdNav thresholds; if collision and success rates fluctuate with the initial hyperparameters or fall to backbone level when the initial threshold is far from the evaluation threshold, the claimed benefit of dynamic thresholding would be refuted. A simpler version would run the Table IV ablation with a grid of deviation and adaptation-rate values and check whether the 0.0146 collision rate persists.","supporting_citations":[{"cited_title":"Training diffusion models with reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the policy-gradient MDP formulation for training diffusion models that the paper adapts."},{"cited_title":"A game-theoretic framework for joint forecasting and planning,","cited_arxiv_id":null,"evidence_quote":"Supplies the game-theoretic forecaster, the collision cost baseline, and the CrowdNav and ETH-UCY collision and success thresholds."},{"cited_title":"Dtpp: Differentiable joint conditional prediction and cost evaluation for tree policy planning in autonomous driving,","cited_arxiv_id":null,"evidence_quote":"State-of-the-art differentiable planning baseline compared on both datasets."},{"cited_title":"Differentiable integrated motion prediction and planning with learnable cost function for autonomous driving,","cited_arxiv_id":null,"evidence_quote":"Differentiable integrated prediction-planning baseline compared on both datasets."},{"cited_title":"Crowd-robot interaction: Crowd-aware robot navigation with attention-based deep reinforce- ment learning,","cited_arxiv_id":null,"evidence_quote":"CrowdNav dataset and its crowd-robot interaction setting used for training and evaluation."},{"cited_title":"You’ll never walk alone: Modeling social behavior for multi-target tracking,","cited_arxiv_id":null,"evidence_quote":"ETH subset of the ETH-UCY benchmark used as the validation scene."},{"cited_title":"Crowds by example,","cited_arxiv_id":null,"evidence_quote":"UCY subsets of the ETH-UCY benchmark used in training."}],"review_version":1}