{"id":"d7cabfd1-725b-44b4-a821-c3bcdbd93eee","arxiv_id":"2501.14678","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":12,"one_line_summary":"An Informer transformer predicts surgical tool-tip position under simulated packet loss and is reported to exceed 90% accuracy on JIGSAWS knot-tying trials.","lead":"Remote robotic surgery needs accurate position prediction when network packets are lost. This paper applies the Informer transformer to predict the surgical robot tool-tip position on the JIGSAWS dataset, claiming over 90% accuracy under simulated packet loss.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq (15)'s position-error term is not computable at inference without label leakage; the reported >90% accuracy may rely on ground-truth positions in the forward pass, invalidating the central claim.","rationale":"The reader's weakest assumption identifies Eq (15) as the critical flaw, and my independent analysis agrees. The paper's central claim is that the Informer framework achieves over 90% prediction accuracy under network impairments. For this claim to be meaningful, the model must not see ground-truth positions during inference. Eq (15) introduces a term in the attention sparsity metric that explicitly depends on the true-minus-predicted position. The text provides no mechanism for computing this term when the true position is unknown, and it does not state that the term is removed at test time. This is not a minor reporting omission: if the term is active, the model is informed of the answer during evaluation; if it is inactive, the paper misdescribes its own architecture. Either way, the reported accuracies cannot be accepted as evidence for the central claim. The other issues the reader noted — undefined accuracy metric, lack of code, absent ablations for the differentiable optimization layer — strengthen the rejection but are secondary. A revision must clarify or remove Eq (15), release code, and define the accuracy metric before the claims can be evaluated. Since my concern matches the reader's and the verdict already reflects this, no adjustment is needed.","tokens_in":12506,"tokens_out":5192,"duration_ms":48525,"concrete_test":"Re-implement or request the authors' code and run the JIGSAWS knot-tying evaluation with Eq (15) modified to set λ1 = 0 (i.e., revert to the original Informer sparsity metric M(qi,K)). Compare the resulting X/Y/Z accuracies and MSE/MAE/RMSE to Table II. If the accuracy drops substantially (e.g., more than a few percentage points), the reported >90% results depend on ground-truth leakage through ex(t). Additionally, inspect the forward pass at inference: if ex(t) is assigned using p(t) from the test labels rather than being zeroed or estimated, the concern is confirmed directly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — Informer-based position prediction exceeds 90% accuracy under packet loss — rests on the evaluation being a genuine prediction task. Eq (15) defines Mpos(qi, K) = M(qi, K) + λ1 ex(t)^T W ex(t), where ex(t) = p(t) − p_hat(t). At inference, p(t) is the target being predicted and p_hat(t) is the model's output before the attention computation. The paper never states that this term is dropped or estimated at test time. If p(t) is read from labels to compute ex(t), the attention mechanism is conditioned on ground truth, so the reported accuracies are not predictive but leaked. If p_hat(t) is instead used, the forward pass becomes circular: ex(t) depends on the model output, which depends on attention, which depends on ex(t). No implementation details, code, or ablation resolve this ambiguity. Because Eq (15) is presented as an integral part of the model's forward pass, the validity of every reported accuracy figure in Section VI is in doubt. This is the load-bearing failure: without a usable, non-leaky definition of the sparsity metric, the experiments do not demonstrate prediction accuracy. The undefined 'accuracy' metric and missing train/test splits compound the problem, but Eq (15) alone is sufficient to undermine the claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes an Informer-based prediction framework for estimating the patient-side manipulator's tool-tip position in remote robotic surgery under network packet loss. Packet loss is simulated with a four-state hidden Markov model, and the authors claim to integrate an optimization problem into the Informer's training via a differentiable optimization layer, modifying the ProbSparse attention sparsity metric to prioritize position error. The method is evaluated on the JIGSAWS knot-tying dataset, reporting over 90% prediction accuracy under various network conditions and claiming superiority over TCN, RNN, and LSTM.","tokens_in":12907,"tokens_out":6285,"duration_ms":57867,"significance":"If the reported results were valid, this would be a practically relevant demonstration of an efficient transformer for a Tactile Internet application, with the O(L log L) complexity argument being a strength. The use of a public dataset and comparison with standard baselines is another positive. However, the central numerical claims are not currently credible: Eq. (15) as written introduces ground-truth leakage into the attention mechanism, the 'accuracy' metric is never defined, the evaluation protocol lacks essential details, and no code or error bars are provided. These issues undermine the paper's headline contribution, although the underlying idea remains potentially salvageable.","major_comments":[{"comment":"The modified sparsity metric M_pos(q_i, K) = M(q_i, K) + λ1 e_x(t)^T W e_x(t), with e_x(t) = p(t) − p_hat(t), is stated without any qualification about its use at inference. At test time, p(t) is the unknown target position, so this term cannot be computed without accessing ground-truth labels. If the term is active during inference, the attention mechanism is conditioned on the target, making the reported accuracies in Section VI the result of label leakage rather than genuine prediction. If the term is intended only for training, this must be stated explicitly, the inference-time forward pass must be defined, and the experiments must be re-run accordingly. As written, the validity of every accuracy figure in Table II and Figure 4 is in question.","section":"Section V-B-b, Eq. (15)"},{"comment":"The paper never defines the 'accuracy' metric used to report values such as 96.68% for the X axis. Without a formula (e.g., 1 − normalized RMSE, threshold-based hit rate, or R²), these percentages are uninterpretable and cannot be compared across models or conditions. The authors should define the metric precisely, report it alongside MSE/MAE/RMSE, and provide uncertainty estimates (e.g., standard deviation over multiple runs or trials).","section":"Section VI-C2, Table II"},{"comment":"The experimental protocol is incomplete. There is no description of the train/test split (e.g., which of the 39 JIGSAWS knot-tying trials are used for training versus testing), how the HMM packet-loss corruption is applied to training and test sequences, the hyperparameters of the Informer and baselines, or the number of runs. The comparison in Table I is therefore not reproducible, and it is unclear whether TCN, RNN, and LSTM were tuned with comparable effort or received identical input features. The authors should provide full implementation details, code, and a clear evaluation protocol.","section":"Section VI"},{"comment":"The claimed 'differentiable optimization layer' is not actually specified. Equations (14), (16), and (17) are weighted loss terms appended to the training objective, not an OptNet-style differentiable optimization layer with a constrained forward pass, KKT conditions, or a projection step. Either the paper should present the actual layer and how constraints are enforced during the forward pass, or it should reframe the contribution as a multi-term training loss with penalty-based regularization. As written, the contribution 'embedding constraints using a differentiable optimization layer' is not supported.","section":"Section V-a"}],"minor_comments":[{"comment":"The heading contains a typo: 'Approch' should be 'Approach'.","section":"Section IV-B"},{"comment":"Equation (10) is written identically to the standard attention formula in Eq. (7), but the text says Q is a sparse matrix containing the top-u queries. The equation should explicitly use the selected sparse queries, e.g., \\bar{Q}, to avoid confusion.","section":"Section IV-B-d, Eq. (10)"},{"comment":"The symbol \\hat{p}(t) is used to denote the corrupted input sequence (set to zero on packet loss), whereas elsewhere in the paper \\hat{p}(t) denotes the predicted position. This notation clash makes the data flow confusing and should be resolved, for example by using p_corrupted(t) for the corrupted input.","section":"Section IV-A, Eqs. (5)-(6)"},{"comment":"The reference list is malformed: reference [2] is truncated ('Accelerating Tactile...'), and reference [3] appears to be merged with the text of reference [4]. The bibliography needs to be cleaned up.","section":"References"},{"comment":"The text says the packet loss pattern is shown over 1000 time steps, while Fig. 4 shows predictions for 200 test time steps; the relationship between these sequence lengths and the JIGSAWS trial lengths should be clarified.","section":"Section VI-C1, Fig. 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an application-oriented preprint, and the central issue is experimental validity rather than the underlying idea. The Eq. (15) leakage concern is severe and must be addressed before any further consideration; the authors should also provide a precise accuracy definition and a complete, reproducible evaluation. The paper may be salvageable if the authors can demonstrate that the attention modification is not leaking ground truth at inference and that the reported gains persist under a properly defined metric and protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper applies the published Informer architecture to PSM position estimation in telesurgery, using a 4-state HMM to simulate packet loss and comparing against TCN, RNN, and LSTM on JIGSAWS. That's a legitimate and relevant extension, and the comparison is useful. The HMM loss model is a reasonable way to generate bursty loss patterns. The complexity analysis is correct: the sparsity modification is constant-time. The soft spots are significant. Eq (15) defines the modified sparsity metric as Mpos(qi,K) = M(qi,K) + λ1 ex(t)^T W ex(t) with ex(t)=p(t)−p_hat(t). At inference, p(t) is the value being forecast. If this term is active during testing, the attention mechanism is conditioned on ground truth, so the reported 90%+ accuracies are not predictive. If it is dropped at test time, the paper never says so. Either way, the empirical claim is unsupported. The paper also never defines the 'accuracy' percentage; Table II reports X/Y/Z accuracy without a formula or error bars. There are no significance tests, no train/test split details, and no ablation isolating the modified sparsity metric from the baseline Informer. The differentiable optimization layer is described but there is no evidence it was implemented or that constraints actually changed the loss. The paper is not incoherent on its own terms—the training-time use of the sparsity modification makes sense—but the inference ambiguity is a load-bearing flaw. I don't see evidence of deliberate fabrication; it looks like a missing statement turned into a circularity. Who is this for? Researchers working on predictive compensation for teleoperation under packet loss. They would get the most out of the HMM packet-loss setup and the Informer implementation details, if those were supplied. My recommendation: send to peer review, but the authors need to clarify Eq (15)'s test-time behavior, define the accuracy metric, provide code or at least a detailed data split, and add ablations. Without those, the central claim has to be treated as unverified. If the clarification reveals leakage, the paper should be rejected; if the term is dropped at inference, a revised version with proper evaluation might be acceptable.","headline":"A relevant application of Informer to telesurgery position prediction, but the undefined accuracy metric and the unresolved ground-truth term in Eq (15) make the >90% claim unsupported.","tokens_in":13395,"tokens_out":2980,"would_cite":false,"duration_ms":27680,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an Informer-based Transformer predicts a surgical robot's tool-tip position with over 90% accuracy under packet loss, outperforming LSTM, RNN, and TCN on the JIGSAWS knot-tying task.","keywords":["Tactile Internet","remote robotic surgery","position estimation","Informer model","packet loss","Hidden Markov Model","JIGSAWS dataset","long-sequence time-series forecasting"],"falsifier":"Run the model's test phase with the true-position term in Eq. (15) replaced by a causal estimate or removed, and compare the resulting X, Y, and Z accuracies with the reported 96.68%, 95.96%, and 90.37%; a large drop would show the published accuracy depends on access to future ground truth.","tokens_in":12309,"feed_emoji":"🤖","tokens_out":8942,"duration_ms":76866,"temperature":0.7,"pith_summary":"The paper is trying to establish that a Transformer-based forecasting model called the Informer can keep the patient-side robot arm of a remote surgical system moving accurately even when the network drops packets. The authors simulate random and bursty packet loss with a four-state Hidden Markov Model, feed the corrupted position stream into the Informer, and report per-axis prediction accuracies of 96.68%, 95.96%, and 90.37% on JIGSAWS knot-tying kinematics. If the claim holds, telesurgery systems could compensate for lost haptic data locally rather than waiting for retransmissions. The paper also claims this framework beats RNN, LSTM, and TCN baselines on the same task while keeping computational cost near $\\mathcal{O}(L \\log L)$.","feed_headline":"Transformer model predicts surgical robot position above 90%","feed_subtitle":"Model copes with packet loss in telesurgery and beats LSTM, RNN, and TCN baselines.","key_machinery":"The load-bearing machinery is the Informer model, a Transformer variant whose ProbSparse attention keeps only the top-$u$ queries selected by a KL-divergence sparsity score, cutting attention cost from $\\mathcal{O}(L^2)$ to $\\mathcal{O}(L \\log L)$. A four-state Hidden Markov Model supplies realistic packet-loss patterns that corrupt the position stream before it reaches the predictor. The paper's modification adds the squared true-minus-predicted position error $e_x(t)^\\top W e_x(t)$ to the sparsity metric, and the optimization problem from Section III is folded into training through loss terms for position error, energy, smoothness, and network robustness.","core_discovery":"The central claim, stated on the paper's own terms, is that the Informer architecture—a Transformer built for long-sequence time-series forecasting—can be adapted to estimate the 3D tool-tip position of the patient-side manipulator accurately enough to absorb network-induced packet loss in Tactile Internet telesurgery. The authors modify the Informer's ProbSparse attention with a position-error term, embed constraints such as energy efficiency and smoothness through a differentiable optimization layer, and augment the input with simulated network conditions. Evaluated on 39 knot-tying trials from the JIGSAWS dataset, the model is reported to achieve over 90% accuracy on all three axes and lower MSE, MAE, and RMSE than LSTM, RNN, and TCN.","pith_inferences":["Inference: the same packet-loss-plus-Informer pipeline could be applied to other telemetry channels, such as orientation, gripper angle, or force, which are equally latency-critical in telesurgery.","Inference: the per-axis accuracy differences suggest a future variant could allocate attention budget dynamically per axis, putting more weight on the fastest-moving coordinate.","Inference: feeding predicted latency and jitter as auxiliary inputs, as the paper proposes, points toward a closed-loop controller where the predictor and the network estimator are trained jointly."],"forward_implications":["If the reported accuracy is reproducible, telesurgery systems can keep the patient-side arm on course during packet loss without relying on retransmission, which would add unacceptable latency.","The $\\mathcal{O}(L \\log L)$ complexity claim means the same model could run on edge hardware near the patient side, not only in a data center.","The accuracy gap between axes, with Z lagging X and Y, suggests future versions should weight attention toward the axis with the fastest dynamics.","The four-state HMM evaluation creates a benchmark protocol that separates burst-loss and random-loss behavior, making model comparisons more informative than a single average loss rate.","Embedding network features such as predicted latency and jitter as auxiliary inputs points toward a predictor that adapts to changing network conditions in real time."],"supporting_citations":[{"why":"Supplies the Informer architecture, ProbSparse attention, distilling, and the $\\mathcal{O}(L \\log L)$ complexity that the paper adapts.","marker":"[9]"},{"why":"Provides the JIGSAWS surgical kinematics dataset used for training and evaluation.","marker":"[10]"},{"why":"Defines the scaled dot-product self-attention baseline whose quadratic complexity the Informer avoids.","marker":"[15]"},{"why":"Supports the generative-style decoder used to make fast multi-step predictions without accumulated error.","marker":"[18]"},{"why":"Provides the four-state Markov packet-loss model used to simulate burst and random network errors.","marker":"[22]"},{"why":"Motivates the differentiable optimization layer through which constraints are embedded in training.","marker":"[25]"}],"fun_headline_variants":["Informer model: 90% accuracy in telesurgery positioning","Telesurgery robot arm prediction hits 90% with Informer","Informer beats old models in surgical robot positioning","Packet loss no match for Informer in remote surgery"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that the model predicts future positions above 90% accuracy depends on a formula that, as written, needs the true position being predicted; if that formula is active at test time, the accuracy figure is not a prediction.","fun_headline_variants_meta":{"raw":{"variants":["Informer model: 90% accuracy in telesurgery positioning","Telesurgery robot arm prediction hits 90% with Informer","Informer beats old models in surgical robot positioning","Packet loss no match for Informer in remote surgery"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1468,"prompt_tokens":922,"completion_tokens":546,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":478}},"tokens_in":538,"tokens_out":546,"duration_ms":5133,"temperature":1.0,"reasoning_tokens":478,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:54:30.850277+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the model's test phase with the true-position term in Eq. (15) replaced by a causal estimate or removed, and compare the resulting X, Y, and Z accuracies with the reported 96.68%, 95.96%, and 90.37%; a large drop would show the published accuracy depends on access to future ground truth.","supporting_citations":[{"cited_title":"(2021, May)","cited_arxiv_id":null,"evidence_quote":"Supplies the Informer architecture, ProbSparse attention, distilling, and the $\\mathcal{O}(L \\log L)$ complexity that the paper adapts."},{"cited_title":"S., Reiley, C","cited_arxiv_id":null,"evidence_quote":"Provides the JIGSAWS surgical kinematics dataset used for training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the generative-style decoder used to make fast multi-step predictions without accumulated error."},{"cited_title":"The Role of Network Packet Loss Modeling in Reliable Transport of Broadcast Audio","cited_arxiv_id":null,"evidence_quote":"Provides the four-state Markov packet-loss model used to simulate burst and random network errors."},{"cited_title":"Zico Kolter","cited_arxiv_id":null,"evidence_quote":"Motivates the differentiable optimization layer through which constraints are embedded in training."}],"review_version":1}