{"id":"27febb62-fcfe-4770-98de-2cb7f3e9c157","arxiv_id":"2607.15656","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A Kalman-cleaned LSTM trained on sensor data reproduces the closed-loop joint motion of a full-size hydraulic excavator over hundreds of seconds, in simulation and on the physical machine.","lead":"This paper trains an LSTM network to act as a digital stand-in for a full-size hydraulic excavator, learning the machine's input-to-motion behavior from sensor data instead of modeling its internal hydraulics. Validated in simulation and on a real excavator, the surrogate tracks long motion trajectories closely enough to serve as an offline testbed for excavation automation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation ground truth is the filter's own output; reported fidelity may measure agreement with a manufactured signal, not the physical excavator.","rationale":"The reader's weakest_assumption identifies the same load-bearing issue: the evaluation ground truth is manufactured by the paper's own consistency-enforcement filter. This is more than a reporting gap; it directly undermines the strongest claim that the surrogate can replace the physical system. The paper explicitly says the filter is not designed to improve estimation accuracy, so there is no evidence that the filtered signal is closer to the true state than the raw measurements. The bucket fusion RMSE of 0.1093 rad is large enough to matter, and using the same processed signal for both training and evaluation makes the high R2 self-referential. The concern is addressable: compare against raw or independent measurements. Because the paper's methodology may still be sound if the filter is accurate, a conditional verdict is appropriate rather than outright rejection. I agree with the reader's CONDITIONAL verdict and recommend no change.","tokens_in":8958,"tokens_out":3720,"duration_ms":38000,"concrete_test":"Hold out a segment of raw inclinometer and gyroscope data from the same test commands, and compute the surrogate's angle/velocity errors against the raw measurements (or an independent high-precision encoder/motion-capture reference) instead of against the Kalman-filtered signal. If angle R2 drops below 0.9 or bucket RMSE remains ~0.1 rad on raw data, the drop-in claim fails. Also report the filter's bias by comparing theta_hat to raw theta in intervals where c_t=1; if bucket bias is systematic, the LSTM is learning that bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract; Conclusion) is that the LSTM is a drop-in surrogate for the physical SY750H. But the target for both training and evaluation is Eq. (4): omega_hat_t = (theta_hat_t - theta_hat_{t-1})/T_s, where theta_hat is the output of the paper's own adaptive Kalman filter (Algorithm 1). Section IV states the filter's objective is 'not to improve estimation accuracy per se, but to enforce physically consistent training data.' Thus the filter can introduce smoothing or bias, and any such distortion becomes the ground truth. The reported bucket fusion RMSE is 0.1093 rad (Section V.C.1) versus ~0.004 rad for other joints, showing the filter can deviate substantially from raw measurements. Since the same filtered signal is used as ground truth in Fig. 4 and Table III, the reported R2 values (e.g., angle R2 > 0.92) measure how well the LSTM reproduces the filter output, not how well it reproduces the physical machine. The 'drop-in surrogate for physical systems' claim therefore rests on an unvalidated premise that the Kalman output is a faithful estimate of true joint state. If the filter distorts the dynamics, the surrogate may be learning a biased or over-smoothed operator that would not match raw sensor readings or independent ground truth.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a system-level surrogate for a four-degree-of-freedom hydraulic excavator using a two-layer LSTM. The excavator is treated as an input–output operator mapping a window of joint states and normalized control commands to joint angular velocities. Training uses a multi-step rollout objective with an optional bias-penalty term. The framework is first validated on a MuJoCo simulation and then on a real SY750H excavator, where an adaptive Kalman filter with consistency detection preprocesses noisy angle/velocity measurements. The surrogate is evaluated autoregressively under held-out control commands over horizons of 90 s (simulation), 382.85 s (no-load real), and 295 s (compound real), reporting RMSE, MAE, and R² for both angular velocity and integrated angle trajectories. The paper claims the learned model can serve as a drop-in surrogate for both the simulator and the physical machine in closed-loop learning and control pipelines.","tokens_in":9175,"tokens_out":3525,"duration_ms":35138,"significance":"If the central claim is correct, the paper addresses a real bottleneck: limited access to heavy hydraulic machines for iterative development of autonomous excavation algorithms. Treating the excavator as a black-box operator and learning an LSTM surrogate from conveniently measured joint states is an appealing and potentially practical approach. The paper has notable methodological strengths: the evaluation is a genuine held-out autoregressive rollout under unseen control commands, the horizons are long (hundreds of seconds), and per-joint metrics are reported for both velocity and integrated angle trajectories. The simulation stage provides a clean sanity check of the operator-learning paradigm. However, the significance is contingent on whether the reported fidelity reflects agreement with the physical machine or only with the paper's own Kalman-filtered signal. As written, the load-bearing 'drop-in surrogate for physical systems' claim is not supported by the evidence presented.","major_comments":[{"comment":"The evaluation ground truth for both training and closed-loop evaluation is the Kalman-filtered angle θ̂ and its numerical derivative ω̂_t=(θ̂_t−θ̂_{t−1})/T_s. Section IV explicitly states that the filter's objective is 'not to improve estimation accuracy per se, but to enforce physically consistent training data.' The reported bucket-joint fusion RMSE of 0.1093 rad (~6.3°) versus ~0.004 rad for the other joints (Fig. 3) shows that the filter can deviate substantially from raw measurements. Because Eq. (4) makes the velocity and integrated-angle targets mutually consistent by construction, the R² values in Table III (e.g., angle R²>0.92) primarily measure how well the LSTM reproduces the filter output, not how well it reproduces the physical machine. The Abstract and Conclusion claim that the surrogate can replace the physical system; this requires validation against raw sensor data or i","section":"§IV-B, Eq. (4), §V-C, Table III"},{"comment":"The paper claims that the bias-penalized multi-step rollout training improves long-horizon stability and that consistency-aware state estimation 'significantly improves the stability and reliability of surrogate learning.' However, no ablation or baseline comparisons are provided. Table I shows that Stage I and the no-load real experiment use H=1, λ_bias=0, while the compound experiment uses H=10, λ_bias=0.3, but there is no comparison of H=1 vs. H=10, no λ_bias sweep, no variant trained without the Kalman preprocessing, and no physics-based or one-step-trained baseline. These ablations are necessary to support the claimed contributions and to rule out the possibility that the reported performance comes mainly from the filter or from the particular training schedule rather than from the proposed method.","section":"§III-B, §V-D, Table I"},{"comment":"All quantitative results are point estimates from a single autoregressive rollout per condition. No error bars, confidence intervals, multiple seeds, or repeated trials are reported. Given the paper's strong claim that the surrogate is a functional replacement for a physical excavator, the absence of any uncertainty quantification is a serious gap. At a minimum, the authors should report statistics over multiple random initializations of the LSTM and, ideally, over multiple held-out test trajectories. This is especially important for the velocity metrics, where some joints already show modest R² values (e.g., boom velocity R²=0.77 in simulation, bucket velocity R²=0.58 in no-load).","section":"§V-A–V-C, Tables II–III"}],"minor_comments":[{"comment":"Equation (2) writes the surrogate as producing \\tilde{x}_{t+1}, but the model output is defined as joint angular velocity \\tilde{y}_t, and the network input includes states and controls. The notation should be made consistent.","section":"§II-B, Eq. (2)"},{"comment":"The table entries are written as 'No-load / Compound' without column headers; this is easy to misread. Consider splitting into two tables or adding explicit subcolumns.","section":"Table III"},{"comment":"The vertical scales for bucket velocity differ markedly between the no-load panel (range about ±5 rad/s) and the compound panel (range about −1 to 0 rad/s), making visual comparison of fidelity difficult. Use consistent scales or clearly annotate the difference.","section":"Fig. 4"},{"comment":"The term 'simulation-to-real' is potentially misleading: the MuJoCo model is used only to validate the methodology, and the real-world model is retrained on real data. No parameters learned in MuJoCo are transferred. Clarify that the paper provides a staged methodology validation, not a true transfer of a trained model.","section":"§III-A, §V"},{"comment":"The thresholds τ_jump, τ_cons, the noise weights R_trust/R_untrust, and the process noise Q are not given numerically and no sensitivity study is provided. At least list the values used and mention how they were selected.","section":"§IV-A, Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The core concern is external validity: the reported metrics are computed against the paper's own Kalman-filtered signal, and Section IV concedes the filter is not designed to maximize accuracy. If the authors can show that the surrogate also tracks raw sensor measurements (or that the Kalman output is validated against independent ground truth), the paper would be substantially stronger. As it stands, the 'drop-in surrogate for physical systems' claim is premature. I recommend major revision rather than rejection because the issue is addressable by additional evaluation and ablations within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the system-level result: a 4-DoF full-size excavator surrogate, trained on Kalman-consistency-cleaned data from a real SY750H, evaluated by long-horizon free rollout in both MuJoCo and the physical machine. That specific combination is not in the cited literature, and the held-out autoregressive evaluation is the right protocol. Angle R2 above 0.9 over 300+ s on most joints is non-trivial and probably useful to anyone doing RL or controls development for heavy machinery without constant access to a machine.\n\nWhat the paper does well: it treats the excavator as an input–output operator rather than pretending to model hydraulics; the staged sim-to-real validation is sensible; and the consistency-aware Kalman preprocessing is a practical answer to the real problem that raw inclinometers and rate gyros contradict each other. The multi-step rollout training with the bias-penalty term is also reasonable.\n\nSoft spots, in proportion: first, the stress-test note is right that the evaluation target is the filter's own output. Eq. (4) differentiates the fused angle, and Section IV explicitly says the goal is not estimation accuracy but consistency. So the R2 numbers measure how well the LSTM reproduces the Kalman-filtered signal, not how well it matches raw sensor truth. The bucket fusion RMSE of 0.1093 rad is an order of magnitude larger than the other joints, which is a red flag that the filter can distort the signal. This is not fatal—if the surrogate is meant to replace the filtered pipeline, it may still be the right object to learn—but it means the abstract's 'drop-in surrogate for physical systems' is too strong. The paper should either validate against an independent ground truth or qualify the claim.\n\nSecond, there are no baselines. No physics model, no one-step LSTM, no without-Kalman variant. That makes it hard to know what the consistency filter and the bias term actually buy you. Third, the compound-condition hyperparameters (H=10, lambda_bias=0.3) were chosen after observing drift on that very condition, and there are no error bars or multiple seeds, so the reported numbers are point estimates. None of these are fatal, but they are all addressable.\n\nThe core approach holds up: the surrogate tracks the system well under free rollout, and the paper is honest about where it had to add tricks to stop drift. I would send this to peer review, because the problem is important, the empirical scope is rare, and a good referee can push the authors to tighten the claims and add the missing comparisons.\n\nFor a reading group: maybe, if someone in the group works on sim-to-real for heavy machinery. I probably would not cite it in my own next paper, but only because it is one step outside my immediate area.","headline":"Solid engineering paper with a real validated result on a full-size excavator, but the 'drop-in replacement for the physical machine' claim outruns the evidence, mostly because evaluation ground truth is the authors' own Kalman-filtered signal.","tokens_in":9796,"tokens_out":1287,"would_cite":false,"duration_ms":15795,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","93E11","93C85"],"pacs":[],"model":"deepseek-v4-flash","headline":"A data-driven LSTM surrogate can stand in for both simulator and physical hydraulic excavator under identical control inputs.","keywords":["hydraulic excavator","digital surrogate","LSTM","system identification","sim-to-real transfer","Kalman filtering","closed-loop behavioral equivalence","autoregressive rollout"],"falsifier":"Run the same autoregressive surrogate in a closed loop with a controller that uses the surrogate's angle predictions to plan or react, and compare task-level outcomes (e.g., digging cycle completion, trajectory tracking error) against the physical excavator under identical conditions. If the surrogate's predictions are used to close a feedback loop and the physical machine diverges or becomes unstable, the claimed drop-in functional equivalence would fail. Alternatively, compare the Kalman-fused angle trajectory against independent high-precision external motion capture on the bucket joint, wh","tokens_in":8732,"feed_emoji":"🚜","tokens_out":1352,"duration_ms":17419,"temperature":0.7,"pith_summary":"This paper tries to establish that a whole hydraulic excavator, with its unmodeled hydraulic dynamics, delays, and sensor noise, can be replaced by a learned input-output operator trained end-to-end from sensor data. The authors argue that an LSTM network, fed with recent joint states and control commands and driven autoregressively, reproduces the machine's angular velocities and long-horizon joint-angle trajectories well enough to serve as a functional stand-in for both a MuJoCo simulator and a real SY750H excavator. They introduce a consistency-aware adaptive Kalman filter to reconcile inconsistent raw angle and angular-velocity measurements before training, and they report high angle-trajectory R-squared values (mostly above 0.9) over free rollouts of several minutes. If correct, this would let control and reinforcement-learning pipelines be developed and tested against a drop-in surrogate rather than against expensive, scarce physical hardware.","feed_headline":"LSTM surrogate stands in for real excavator","feed_subtitle":"A data-driven operator learned on Kalman-cleaned sensor data reproduces long-horizon motion of a hydraulic excavator.","key_machinery":"The central object is the input-output operator formulation (Eq. 2), where the excavator is treated as a black-box map from recent joint states and control commands to future states, approximated by an LSTM with a 30-step history window. The supporting mechanism is the consistency-aware adaptive Kalman filter (Algorithm 1) that fuses noisy angle measurements with integrated angular velocity, adaptively trusting either the angle observation or the velocity integration based on jump detection and window-consistency checks. The filter's output angle is numerically differentiated (Eq. 4) to produce the angular-velocity training target, enforcing kinematic consistency between the angle and veloci","core_discovery":"The central claim is behavioral equivalence under closed-loop autoregressive evaluation: given the same normalized joint-level control commands, a two-layer stacked LSTM (hidden size 128, input window of 30 steps) that outputs joint angular velocities can be rolled out freely to reproduce the excavator's closed-loop response. In simulation, angle-trajectory R-squared exceeds 0.94 over 90 s. On the real machine, the surrogate achieves angle-trajectory R-squared above 0.92 in no-load conditions and mostly above 0.73 in compound digging operations over 295-383 s, while matching both short-term velocities and long-horizon integrated angles. The paper further claims that its consistency-aware Kal","pith_inferences":["The strongest version of the claim—that the surrogate is a drop-in replacement for the physical machine—is only as strong as the evaluation protocol: the tests use fixed control inputs and recorded trajectories, not closed-loop control that reacts to the surrogate's own outputs, so true policy-in-the-loop equivalence is not yet demonstrated.","Because the training target is the numerically differentiated Kalman-filtered angle, any smoothing or bias introduced by fusion (most visibly on the bucket joint, with fusion RMSE 0.1093 rad versus under 0.012 rad elsewhere) becomes part of the ground truth; the reported fidelity partly measures agreement with a filtered signal rather than with raw physical measurements.","A natural next experiment would be to run a learning-based controller trained on the surrogate directly on the physical machine and measure task success, which would test whether the surrogate preserves the input-output behavior required for control, not just open-loop reproduction.","The paper's framework suggests that the separator between simulation and reality is not the physics engine but the data consistency pipeline; if so, the MuJoCo stage may be replaceable by any reasonably realistic simulator, or even by recorded real data alone."],"forward_implications":["If the surrogate truly reproduces closed-loop behavior, control policies and reinforcement-learning training loops can be run against the surrogate instead of the physical excavator, greatly increasing iteration speed and reducing hardware wear and risk.","The same operator-learning recipe could apply to other hydraulically actuated heavy machinery with limited sensor instrumentation, provided the consistency-enforcement pre-processing is adapted to their sensors.","Long-horizon autoregressive stability, which the paper achieves through a bias-penalized multi-step loss, becomes a practical way to assess whether a learned dynamics model is trustworthy for closed-loop use.","A single learned model could serve both simulation and reality, potentially absorbing simulator-specific corrections and real-world effects into one data-driven operator, simplifying transfer.","The approach sidesteps explicit system identification of hydraulic parameters, meaning it could work even when internal dynamics are unknown or poorly parameterized."],"fun_headline_variants":["LSTM surrogate trained in sim reproduces real excavator motion","Sim-to-real LSTM predicts excavator trajectories long-term","Surrogate excavator model matches closed-loop motion in sim and real","Kalman-filtered LSTM surrogate mimics excavator behavior"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole evaluation rests on treating the angular velocity computed as the numerical derivative of the Kalman-fused angle as the ground truth; if that fused signal is systematically biased or over-smoothed relative to the excavator's actual physical motion, the reported 'fidelity' measures agreement with the filter's manufactured signal, not with the real machine.","fun_headline_variants_meta":{"raw":{"variants":["LSTM surrogate trained in sim reproduces real excavator motion","Sim-to-real LSTM predicts excavator trajectories long-term","Surrogate excavator model matches closed-loop motion in sim and real","Kalman-filtered LSTM surrogate mimics excavator behavior"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000514,"raw_usage":{"total_tokens":2310,"prompt_tokens":695,"completion_tokens":1615,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":1545}},"tokens_in":439,"tokens_out":1615,"duration_ms":11174,"temperature":1.0,"reasoning_tokens":1545,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T22:39:29.696749+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same autoregressive surrogate in a closed loop with a controller that uses the surrogate's angle predictions to plan or react, and compare task-level outcomes (e.g., digging cycle completion, trajectory tracking error) against the physical excavator under identical conditions. If the surrogate's predictions are used to close a feedback loop and the physical machine diverges or becomes unstable, the claimed drop-in functional equivalence would fail. Alternatively, compare the Kalman-fused angle trajectory against independent high-precision external motion capture on the bucket joint, wh","supporting_citations":[],"review_version":1}