{"id":"c1430eed-33e4-46b5-947f-5ce6fcb9668d","arxiv_id":"2502.05996","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"In a Simulink drone simulator, TD3 with curriculum learning and acceleration feedback achieved lower positional error and higher success than DDPG and than TD3 without curriculum.","lead":"The paper trains two deep reinforcement learning algorithms, DDPG and TD3, in a simulated drone that navigates waypoints for additive manufacturing tasks. It reports that TD3 with curriculum training and acceleration-based observations handles changing payloads more reliably than the alternatives tested.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 94% mass-adaptation result is not cleanly attributable to acceleration feedback: Eq. (3) already lists ax,ay,az while §V-C claims the observation space was 'expanded' to include them, with no controlled ablation isolating the change.","rationale":"The reader's weakest assumption ('clean accelerations in Simulink, so 94% may not transfer') is a transfer concern that would matter if the paper claimed real-world deployment; the paper only claims a simulated control approach and lists real-world implementation as future work. The more direct threat to the central claim is internal: the causal story for the headline 94% result is not established within the simulation because the reported state definition and the reported observation-space modification contradict each other and no ablation isolates the acceleration terms. I agree with the reader's overall conditional verdict: the paper is a clear, self-contained empirical study with a plausible but unverified central effect; the missing ablations, seeds, and code release prevent a stronger verdict. The recommended check is a controlled observation-space ablation with matched seeds. This does not contradict the reader's verdict; it sharpens the condition.","tokens_in":9768,"tokens_out":4988,"duration_ms":50239,"concrete_test":"Re-run the mass-variation experiment with two TD3 agents that are identical in every respect—network size, hyperparameters, random seeds, total environment steps, and curriculum schedule—differing only in whether the observation vector includes ax, ay, az. Use the actual Simulink model to log the observation vector fed to the agent, and verify whether Eq. (3) was used in the 94% run and which reduced vector was used in the 26–29% run. Report per-seed success rates and means over at least 5 seeds; if the success gap collapses or overlaps within seed noise, the acceleration attribution in Section V-C is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim is that adding x/y/z accelerations to the observation space lifts success under variable mass from 26–29% to 94% (Section V-C). This claim is load-bearing because the abstract and conclusion use it to argue that TD3 'consistently' balances accuracy and success under payload changes. The experiments, however, do not isolate the observation-space change. Eq. (3) defines the state vector as including ax,t, ay,t, az,t for the proposed framework, but Section V-C says the observation space 'was expanded to include the drone's accelerations in the x, y and z directions.' If the earlier agents (DDPG, TD3, curriculum) were trained without accelerations, their state representation contradicts Eq. (3); if they already had accelerations, the reported improvement from 'expanding' has no well-defined baseline. The pre/post numbers are also inconsistent: the same condition is reported as 450/26% and 454/29%. Because no training seeds, environment steps, or error bars are reported, the 94% could arise from the retraining procedure, additional experience, or run-to-run variance rather than the acceleration terms. This undermines the paper's most novel and emphasized result, independent of the separate real-world transfer question.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a deep reinforcement learning (DRL) framework for multirotor waypoint navigation in additive manufacturing tasks. The authors model the problem as an MDP with a state vector that includes accelerations, a three-action control space (roll, pitch, thrust), and an exponential proximity reward, and they compare DDPG with TD3 in a MATLAB Simulink UAV simulator. They report that TD3 outperforms DDPG on single-waypoint navigation (average positional error 1.804e-2 m vs. 5.643e-2 m; success 96% vs. 82%), that curriculum learning improves TD3's multi-waypoint success from 66% to 87%, and that adding x/y/z accelerations to the observation space raises TD3's success under variable mass from 26-29% to 94%. The paper concludes that TD3 with curriculum learning and acceleration feedback is a promising control approach for simulated UAV-based additive manufacturing.","tokens_in":10051,"tokens_out":2675,"duration_ms":30742,"significance":"If the reported results are reproducible, the paper would provide a useful empirical comparison of DDPG and TD3 for a simulated UAV control task with payload variation, and the curriculum-learning result would be a reasonable incremental contribution. The paper has concrete strengths: it reports quantitative metrics (cumulative reward, positional error, precision, success ratio), provides a hyperparameter table, uses standard off-policy algorithms, and documents the simulation setup in enough detail to be partially reconstructed. However, the most emphasized result — that acceleration feedback enables adaptation to mass variation — is not supported by a controlled ablation, and the absence of multiple training runs and statistical tests weakens the claims of consistent and significant improvement. The real-world transfer claim is also limited by the absence of process noise and the use of near-ideal acceleration signals from the simulator.","major_comments":[{"comment":"The central mass-adaptation result is not cleanly attributable to adding accelerations, because the baseline condition is not well defined. Eq. (3) already lists ax,t, ay,t, az,t as part of the proposed state vector, but §V-C states that the observation space 'was expanded to include the drone's accelerations in the x, y and z directions.' If the curriculum-trained baseline already used Eq. (3), there was no expansion; if it did not, then the baseline contradicts Eq. (3). The paper reports inconsistent pre-retraining numbers for the same condition (average cumulative reward 450 with 26% success, then 454 with 29% success), and no ablation is provided that keeps all other factors fixed while toggling only the acceleration terms. With no random seeds or error bars, the improvement from 26-29% to 94% cannot be separated from retraining, additional experience, or run-to-run variance. This issue is load-bearing because the abstract and conclusion use the 94% result to claim that TD3 'consistently' balances accuracy and success under payload changes.","section":"§V-C and Eq. (3)"},{"comment":"The claim that TD3's average positional error is 'significantly lower' than DDPG's is not supported by any statistical test or repeated-run evidence. Tables II, III, and IV report single values for each condition, with no number of seeds, no standard errors, and no confidence intervals. Since DRL training is stochastic, a single run cannot establish significance. Additionally, the claim that TD3 'consistently' balances stability, accuracy, and success is not supported by the presented experiments: DDPG is evaluated only on the single-waypoint task and is not tested under curriculum learning, mass variability, or multi-waypoint navigation, so no cross-condition consistency comparison between the algorithms is actually made.","section":"§V-A and Tables II-IV"},{"comment":"The paper's transferability claim is not supported by the simulation model used. The text explicitly states that only measurement noise is added, with no process noise, and that noise is injected after reading the state from a deterministic multirrotor model. Thus the accelerations in Eq. (3) are an almost noiseless, delay-free proxy for mass through F = ma. On a real drone, accelerometer measurements are noisy, delayed, and coupled with attitude dynamics, so the 94% success under variable mass would likely not transfer. The paper should either state this limitation clearly as a simulation-only result or add a sensor model with realistic acceleration noise, delay, and process noise before claiming progress toward 'robust, autonomous drone control in additive manufacturing.'","section":"§II, §III-A, and §V-C"}],"minor_comments":[{"comment":"The text says the success ratio rose 'from 29% to 94%,' but the same condition is earlier given as 26%, and the average cumulative reward is given as 450 and then 454; the inconsistent baseline numbers should be reconciled.","section":"§V-C, Fig. 5"},{"comment":"The reward function rp = wp exp(-||p - p_desired||^2) is described as convex, but a negative exponential of a squared Euclidean norm is not convex over the whole space; this should be reworded or replaced with a precise geometric description.","section":"§III-A, Eq. (5)"},{"comment":"The curriculum stage C1 is defined with reward R(s,a) = -||p_target - p_current||_2, which is inconsistent with the exponential reward in Eq. (5); the authors should clarify which reward is used in the basic navigation stage.","section":"§III-B, Eq. (6)"},{"comment":"There are several typographical and formatting errors, including 'randomaly' in §V-C, inconsistent spacing in 'UA Vs' and 'UAV-based', and a duplicated row in the Target Policy Smooth Model block of Table I (two 'Std. Dev.' entries).","section":"General"},{"comment":"Table III is captioned 'Training Results' but the surrounding text and Fig. 4 describe test results; the caption should be corrected to indicate whether the numbers are from training or from the 100-trial test phase.","section":"§V-B, Table III"}],"recommendation":"major_revision","confidential_remarks":"The manuscript relies heavily on self-citations for background statements ([11]-[15], [19] appear multiple times in the introduction), and some of those references are not clearly connected to the claims they support. This is not by itself disqualifying, but the authors should review whether each citation is necessary and whether the paper would benefit from more independent references for the DRL and UAV control background. The main concern, however, is the missing ablation and statistical rigor around the headline mass-adaptation result; I would like to see a revised version that either supplies the controlled comparison or honestly reframes the claim as a preliminary simulation study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a competent application-style paper: TD3 vs DDPG on simulated multirotor waypoint navigation, with curriculum learning and acceleration-augmented observations when mass varies. The MDP setup and curriculum stages are clearly described, and the 100-trial test protocol is a reasonable way to compare agents. If the numbers are reproducible, the qualitative takeaway—TD3 with curriculum and acceleration feedback handles simulated payload changes well—is a legitimate datapoint for drone-based additive manufacturing. It does not break new algorithmic ground, and the authors do not claim it does.\n\nThe soft spots are real but not fatal. The biggest one is the mass-adaptation comparison in Section V-C. The text says the observation space was \"expanded\" to include x/y/z accelerations, but Eq. (3), which defines the proposed state, already includes them. That leaves it ambiguous what state the earlier agents used, so the 26-29% to 94% success jump is not cleanly attributable to adding accelerations. The pre/post rewards are also inconsistent in the same paragraph (450 vs 454 for what looks like the same condition). No random seeds or error bars are reported, and the word \"significantly\" appears without a statistical test. DDPG is dropped after the baseline, so the claim that TD3 \"consistently\" outperforms it across the harder tasks is not actually tested. The noise model is also generous: measurement noise is added after reading a deterministic simulator state, so the accelerations carry almost ideal mass information; that weakens any real-world transfer claim.\n\nThe citation pattern is a separate minor issue: several background citations point to the authors' own prior work, but the central empirical comparison does not depend on those references, so I do not see circularity. The environment is proprietary and unreleased, which limits independent verification, but that is common for Simulink-based RL studies and not disqualifying.\n\nWho gets value from this? Researchers working on RL-based UAV control, especially those considering curriculum learning or acceleration terms for payload variation. It deserves a serious referee: the topic is relevant, the presentation is clear, and the flaws are fixable. My recommendation is to send it to peer review as a conditional accept, requiring a cleaner ablation of the acceleration observation, seeds or error bars, and either the code or a detailed environment specification.","headline":"A useful but incremental Simulink study of TD3 plus curriculum learning for UAV waypoint navigation; the central mass-adaptation claim is plausible but the observation-space ablation is not cleanly isolated.","tokens_in":10576,"tokens_out":1351,"would_cite":false,"duration_ms":16682,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A reinforcement-learning controller can navigate a simulated drone to waypoints with centimeter accuracy and adapt to changing mass.","keywords":["deep reinforcement learning","multi-rotor UAV control","additive manufacturing","waypoint navigation","curriculum learning","twin delayed deep deterministic policy gradient","mass adaptation","simulation"],"falsifier":"Retrain the same TD3 agent with the acceleration observations replaced by realistic accelerometer signals (additive zero-mean noise with the sensor's variance and a one-step delay) and test under variable mass; if the success rate falls well below 94%, the acceleration-based mass adaptation is an artifact of the clean simulator signal. A direct flight test with a changing payload would settle the question.","tokens_in":9568,"feed_emoji":"🚁","tokens_out":5036,"duration_ms":45772,"temperature":0.7,"pith_summary":"This paper argues that a deep reinforcement learning agent can control a multi-rotor drone performing waypoint navigation for additive manufacturing, and that two design choices make the control robust to the mass changes that occur during material deposition. The authors compare DDPG and TD3 in a MATLAB Simulink simulation and report that TD3 is the better controller, reaching a fixed waypoint with an average positional error of 1.804e-2 m and a 96% success rate versus 5.643e-2 m and 82% for DDPG. They show that a curriculum that gradually increases task complexity raises TD3's multi-waypoint success from 66% to 87%. Their central empirical claim is that adding x/y/z accelerations to the observation space lets the agent infer mass variations through F = ma, raising success under variable mass from 26-29% to 94%. If correct, the result is a scalable simulation-to-reality path for autonomous aerial material deposition.","feed_headline":"RL-trained drone hits waypoints 96% of the time in simulation","feed_subtitle":"TD3 with acceleration feedback adapts to changing mass, lifting success from 29% to 94%.","key_machinery":"The load-bearing object is the observation vector of equation (3), in particular the inclusion of the drone's accelerations $a_{x,t}, a_{y,t}, a_{z,t}$. Because the deposition force is known ($F = 2.135 \\times 10^{-2}$ N) and the simulator's dynamics are deterministic, acceleration is a direct readout of the drone's current mass via $F = ma$; the paper claims this allows the policy to adapt its thrust commands when mass changes. The other machinery is the curriculum schedule $C(t) \\in \\{C_1, C_2, C_3, C_4\\}$, which progressively increases task complexity and stabilizes training, and the TD3 algorithm itself, whose twin critics and delayed policy updates reduce the overestimation bias that harms DDPG.","core_discovery":"The core discovery is that a TD3 policy trained in a Simulink multirotor simulator, with roll, pitch, and thrust as actions and a reward that decays exponentially with distance to the target, can navigate waypoints with centimeter-level accuracy, and that its robustness to mass variability depends on the agent being able to observe its own accelerations. The authors model the control problem as an MDP whose state vector includes the drone's x/y/z accelerations, positional error to the target, velocities, orientation angles, and height. Curriculum learning stages the training from static waypoints to dynamic waypoints, variable mass, and external disturbances. The paper reports that without acceleration feedback the curriculum-trained agent succeeds only 26-29% of the time under variable mass, and with acceleration feedback the success rate reaches 94%, with average positional error dropping from 0.2688 m to 0.09857 m.","pith_inferences":["The acceleration-feedback result suggests a general recipe: when an unknown physical parameter (here, mass) is the only hidden variable, exposing a direct measurement of it in the observation space can convert a partially observable control problem into a nearly observable one; the same trick might apply to other variable-payload aerial tasks.","On real hardware, the gap between the 94% result and reality will likely be dominated by accelerometer noise and transport delay; a natural extension is to train with a realistic IMU model and measure how much accuracy degrades.","The paper does not isolate why TD3 beats DDPG beyond citing overestimation bias; a controlled ablation holding all else fixed would test whether the advantage is algorithmic or due to exploration noise differences.","The deposition-force model used, a constant reaction force of 2.135e-2 N, is small relative to a typical drone's thrust; the mass variability claim may scale only if the force and mass range reflect a real extruder, so stress testing with broader mass ranges would clarify practical limits."],"forward_implications":["A TD3 policy trained in simulation can serve as the control layer for waypoint navigation in drone-based additive manufacturing, reaching targets with average positional error below 2 cm in the simulator.","Curriculum learning is a viable way to train a single policy that handles static waypoints, dynamic waypoints, variable mass, and external disturbances without retuning.","Putting raw acceleration measurements into the observation space is a simple, effective way to make a learned controller aware of mass changes, at least when the measurements are clean.","The framework is extensible in the paper's own terms: the same MDP, reward, and curriculum can be carried toward real-world implementation, which the authors state as future work."],"supporting_citations":[{"why":"Supplies the curriculum learning method used to stage task complexity.","marker":"[22]"},{"why":"Provides the multirotor dynamics model on which the Simulink simulation is built.","marker":"[25]"},{"why":"Gives the material density, nozzle diameter, and flow velocity used to compute the deposition force F = 2.135e-2 N.","marker":"[26]"},{"why":"Motivates the exponential reward-shaping function r_p used for positional control.","marker":"[27]"},{"why":"Reference implementation of DDPG used in the comparison.","marker":"[29]"},{"why":"Reference implementation of TD3 used in the comparison.","marker":"[30]"},{"why":"Provides the experience replay mechanism that the off-policy training relies on.","marker":"[31]"},{"why":"Supplies the weight initialization scheme for actor and critic networks.","marker":"[32]"}],"fun_headline_variants":["TD3 with acceleration feedback lifts drone success from 29% to 94%","Acceleration feedback is key to RL drone's 94% success under mass variability","RL drone with acceleration sensing reaches 94% waypoint success for variable payloads","Simulation shows TD3 drone control robust to mass changes: 29% to 94% success"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that acceleration feedback is what enables mass adaptation assumes the simulator gives the agent clean, near-ideal accelerations; on a real drone, accelerometer readings are noisy, delayed, and mixed with attitude dynamics, so the 94% success under variable mass would likely not transfer directly.","fun_headline_variants_meta":{"raw":{"variants":["TD3 with acceleration feedback lifts drone success from 29% to 94%","Acceleration feedback is key to RL drone's 94% success under mass variability","RL drone with acceleration sensing reaches 94% waypoint success for variable payloads","Simulation shows TD3 drone control robust to mass changes: 29% to 94% success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000883,"raw_usage":{"total_tokens":3781,"prompt_tokens":882,"completion_tokens":2899,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":2808}},"tokens_in":498,"tokens_out":2899,"duration_ms":21888,"temperature":1.0,"reasoning_tokens":2808,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T17:06:52.656185+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same TD3 agent with the acceleration observations replaced by realistic accelerometer signals (additive zero-mean noise with the sensor's variance and a one-step delay) and test under variable mass; if the success rate falls well below 94%, the acceleration-based mass adaptation is an artifact of the clean simulator signal. A direct flight test with a changing payload would settle the question.","supporting_citations":[{"cited_title":"A survey on curriculum learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the curriculum learning method used to stage task complexity."},{"cited_title":"”Trajectory Generation and Control for Precise Aggressive Maneuvers with Quadrotors.” The International Journal of Robotics Research , 2012, pp","cited_arxiv_id":null,"evidence_quote":"Provides the multirotor dynamics model on which the Simulink simulation is built."},{"cited_title":"Aerial additive manu- facturing with multiple autonomous robots,","cited_arxiv_id":null,"evidence_quote":"Gives the material density, nozzle diameter, and flow velocity used to compute the deposition force F = 2.135e-2 N."},{"cited_title":"Challeng- ing common assumptions in convex reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Motivates the exponential reward-shaping function r_p used for positional control."},{"cited_title":"A novel DDPG method with prioritized experience replay,","cited_arxiv_id":null,"evidence_quote":"Reference implementation of DDPG used in the comparison."},{"cited_title":"Real-time au- tonomous residential demand response management based on twin delayed deep deterministic policy gradient learning,","cited_arxiv_id":null,"evidence_quote":"Reference implementation of TD3 used in the comparison."},{"cited_title":"Lin, ”Self-improving reactive agents based on reinforcement learning, planning, and teaching,” Machine Learning , vol","cited_arxiv_id":null,"evidence_quote":"Provides the experience replay mechanism that the off-policy training relies on."}],"review_version":1}