{"id":"dfe87fdf-2981-45e2-83fe-0f7bfe06edfc","arxiv_id":"2505.10224","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Force, torque, and pose recordings from a cobot pressing cockpit controls allow CNNs to classify actions as success or failure, with Grad-CAM heatmaps pointing to the signal regions that drove the decision.","lead":"A team from Politecnico di Milano and TXT E-TECH trained neural networks to judge whether a robot's presses and pulls on aircraft cockpit controls worked correctly, using force, torque, and position signals. The approach is meant to automate aircraft certification tests and explain why a test failed.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Label quality and simulator fidelity are the load-bearing premises; the paper itself flags the knob augmentation as data-snooping, and the 30 Hz Butterworth cutoff is tuned on the test metric, so the F1-scores likely overstate real-world feasibility.","rationale":"The reader's weakest_assumption is precisely the label-ground-truth and simulator-fidelity premise, and my independent reading of Section 3 and Section 6 lands on the same load-bearing concern. The paper is a feasibility study, not a deployment claim, and it contains explicit caveats about setup-specific tuning and future real-cockpit validation that match the CONDITIONAL verdict. I considered whether the lack of uncertainty estimates or the absence of baseline comparisons could be the most load-bearing issue, but those are secondary: even with perfect reporting, the label and simulator premises are what would make the F1-scores mean what the paper claims they mean. The concrete tests I propose would directly settle this: a kappa-based label audit tests the ground-truth premise without requiring access to real aircraft, and a transfer test on a physical cockpit or higher-fidelity simulator tests the representativeness of the simulator's force responses. Both are feasible for the authors, who have access to the ARTO setup and the certified simulator. The paper itself flagging the knob data-snooping risk and the threshold heuristic is in-scope self-reported limitation evidence; it strengthens rather than weakens the concern. Verdict remains CONDITIONAL because the concerns are addressable and the current evidence is plausibly sufficient for a feasibility claim, but not for acceptance as a validated approach.","tokens_in":8418,"tokens_out":2284,"duration_ms":21342,"concrete_test":"Perform a label-quality audit and a simulator-transfer check. First, have two independent annotators re-label a stratified random sample of at least 300 recorded actions using only functional outcomes (e.g., multimeter continuity on the switch, detent position, or state readback from the cockpit simulator), and compute Cohen's kappa between the script-assigned labels and each annotator. If kappa is below 0.9, the reported F1-scores are partly classifying label noise rather than functional success. Second, record a small transfer set of at least 50 actions per component on a physical A320 cockpit or a higher-fidelity simulator with the same UR5e and tooling, and evaluate the already-trained 1D-CNN models without any retuning of the preprocessing constants (30 Hz cutoff, energy thresholds).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central feasibility claim rests on supervised labels being correct ground truth for successful functional actuation, and on simulator wrench behavior being representative of real cockpits. The paper states labels are 'assigned via script' (Section 3) and 'serves as ground truth for classification'; if script assignment conditions differ from functional outcomes, F1 measures label agreement, not validation success. No inter-rater reliability, label-consistency audit, or independent success criterion (e.g., switch state, electrical continuity) is reported. Additionally, the preprocessing pipeline is tuned on the evaluation metric: the Butterworth cutoff 'was determined by running CNN models and comparing their average classification accuracy', and the transient-detection thresholds are 'determined heuristically, posing a potential risk of overfitting to this specific setup'. The knob result is explicitly flagged by the authors as potentially data-snooping. With 4330 manually labeled actions across six components, class imbalances, and augmentation, the reported 95-98% F1-scores do not yet establish feasibility on real aircraft components. The paper itself acknowledges this in Section 6, noting the algorithms are 'well-suited to this particular robot configuration and its embedded Force Torque Sensor' and that further data from simulators and real cockpits is needed. This internal limitation statement is consistent with the reader's concern and supports a CONDITIONAL rather than ACCEPT verdict.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a force-driven validation framework for collaborative robots in automated avionics testing. The system uses a UR5e cobot with an embedded force-torque sensor to interact with cockpit components in an A320 simulator. Force, torque, and end-effector pose signals are recorded, preprocessed with a Butterworth filter and an energy-based transient detection algorithm, and then fed into 1D-CNN, 2D-CNN (using CWT scaleograms), and hybrid architectures. The models classify each action as Success or Fail and Grad-CAM provides visual explanations. Experiments on 4330 manually labeled actions across six components report F1-scores of roughly 95-98% for 1D-CNNs on most components, with the knob action requiring augmentation to reach 95.2%. The authors conclude that the approach is feasible for the specific robot configuration and call for more data from diverse simulators and real cockpits.","tokens_in":8666,"tokens_out":3510,"duration_ms":34213,"significance":"If the reported performance is trustworthy, the paper contributes an interpretable, wrench-based validation method that could reduce manual effort and increase reliability in avionics certification testing. The strength of the work lies in its breadth: it compares several network architectures, integrates 1D Grad-CAM for time-series interpretability, and involves a realistic A320 simulator setup. The authors also make the code available and are transparent about many limitations, including the risk of data-snooping in the knob augmentation. However, the evaluation is hampered by circularity in preprocessing selection and by an unclear labeling protocol, both of which directly affect whether the reported F1-scores reflect genuine functional validation. As a feasibility study the contribution is modest but plausible; the conclusions are currently stronger than the evidence supports.","major_comments":[{"comment":"The Butterworth cutoff frequency (30 Hz) is selected by 'running CNN models and comparing their average classification accuracy' (Section 3). If this comparison uses the same test split from which the final F1-scores are reported, the evaluation is circular and the reported 95-98% scores are optimistically biased. The manuscript must describe the model-selection protocol (e.g., nested cross-validation, tuning on a validation subset, or a separate development set) and report the sensitivity of results to nearby cutoff values.","section":"Section 3 (Butterworth filter cutoff)"},{"comment":"The dataset description is internally inconsistent: the text first says 'a label is assigned via script' and later refers to 'over 4330 manually labeled actions.' If the labels are assigned programmatically based on, say, encoder readings or commanded state, the paper should define the assignment rule and explain how it corresponds to functional success or failure (e.g., switch state, electrical continuity). If the labels are manually assigned, inter-rater reliability or an independent audit should be reported. Without a precise label definition, the F1-scores only measure agreement with the script/manual labels, not the validity of the action.","section":"Section 3 (ground-truth labels)"},{"comment":"Table 1 reports a Knob F1-score of 95.2% after augmentation, while Section 6 states that 'the single exception is the knob task, where an unbalanced dataset constrains performance; however, an F1-score exceeding 78% is still achieved.' This discrepancy appears to reflect the authors' downweighting of the augmented result due to suspected data-snooping, but the paper never explicitly retracts or flags the Table 1 value in the conclusion. The contradictory claims must be reconciled: which result is the actual claimed outcome for the knob task, and what is the basis for trusting one over the other?","section":"Section 5.1 and Section 6 (Knob result inconsistency)"},{"comment":"The transient detection thresholds (0.38, 0.22, 0.12 of max energy) and the window lengths (300, 800 samples) are described as 'determined heuristically, posing a potential risk of overfitting to this specific setup and force controller settings.' Since these parameters are part of the feature extraction and were presumably chosen with knowledge of classification outcomes, they should be treated as hyperparameters. The paper does not report how varying these thresholds affects the final classification performance, nor does it provide an alternative, more principled selection criterion. At minimum, a sensitivity analysis is needed to gauge the robustness of the reported F1-scores to these choices.","section":"Section 3 (energy-based transient thresholds)"}],"minor_comments":[{"comment":"There are typographical errors, including 'traslation' in Section 3 and 'nternational' in reference [15]; these should be corrected.","section":"General (typos)"},{"comment":"The column heading 'Element' in Table 1 appears to denote the action type; please rename it to 'Action' for clarity.","section":"Table 1 (column heading)"},{"comment":"The description of when Standard Scaling is applied versus not is scattered; a concise summary of the normalization choice for each action and input channel would improve reproducibility.","section":"Section 5 (normalization details)"},{"comment":"The Grad-CAM visualizations are presented qualitatively. Since interpretability is a stated contribution, the paper would benefit from a quantitative evaluation (e.g., localization metrics, perturbation tests, or expert agreement) to show that the highlighted regions correspond to meaningful action features.","section":"Grad-CAM evaluation"},{"comment":"The authors provide a GitHub URL, but it is unclear whether the dataset itself is included or only the code. Please clarify whether the data can be accessed for reproducibility.","section":"Code and data availability"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a useful feasibility study in an application area that is not well covered in the literature. However, the evaluation's validity is currently undermined by (i) the selection of a preprocessing hyperparameter using the classification metric without a clear train/validation/test protocol, (ii) an ambiguous labeling protocol that conflates script assignment with manual labeling, and (iii) an inconsistent treatment of the knob result. These issues are addressable with additional experiments and a more careful description, so I believe a major revision is appropriate rather than rejection. I would also note that the paper might be better framed as a systems paper rather than a deep-learning method paper, as the novelty is primarily in the application and the integration of XAI."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a straightforward feasibility study, and a fairly honest one. The new thing is the ARTO dataset: over 4330 labeled actions across six cockpit components, plus a per-action benchmark of 1D-CNN, 2D-CNN, and hybrid variants, and a 1D Grad-CAM adaptation that is genuinely useful for diagnosing where the model looks. The methods themselves are established—CNN on force signals, CWT scaleograms, Grad-CAM—but the domain application and the dataset are new, and the code is on GitHub. That counts for something.\n\nThe paper does several things well. The preprocessing pipeline is described in enough detail to reproduce, the authors explicitly admit when they tuned constants on test accuracy (the Butterworth cutoff, the energy thresholds), and they flag the knob augmentation as potential data-snooping. Section 6 is appropriately restrained: they say the algorithms are tailored to this robot and sensor, and that broader validation needs more data. That internal limitation statement matches what the evidence supports. The use of a held-out test split and per-action results is good practice.\n\nSoft spots, in proportion. The labels are \"assigned via script\" and treated as ground truth, but there is no independent success criterion (e.g., switch state or electrical continuity) and no inter-rater or consistency check. That is the load-bearing premise, and it is only partially audited. The simulator fidelity question is real but not disqualifying for a feasibility claim—nobody expects an A320 simulator to match a real cockpit at this stage. The preprocessing tuning does introduce circularity, but the held-out split gives some independence, and the authors openly disclose the tuning. The lack of confidence intervals on the F1-scores is a minor issue; with per-action sample counts not given, it is hard to know how tight those 95–98% numbers actually are.\n\nThe citation pattern looks fine—relevant prior work is cited, and the authors do not overstate novelty. No invented entities, no hidden dependencies.\n\nWho is this for? People working on force-based robot skill validation, automated aerospace testing, and applied interpretable ML. A reader will get a clear, reproducible baseline for this kind of wrench-based classification, together with an honest discussion of where the approach falls short.\n\nRecommendation: send it to peer review. It is a solid engineering study with a public dataset and code, and the limitations are addressable in revision: add label consistency checks, report uncertainty, and clarify what the script-assigned labels actually measure. My verdict is conditional accept rather than accept—but it is a serious, worth-reading paper.","headline":"A competent, honest feasibility study for force-based cobot action validation in a cockpit simulator; the central claim holds for the specific setup, and it deserves a serious referee despite known preprocessing tuning and label-quality caveats.","tokens_in":9205,"tokens_out":1318,"would_cite":false,"duration_ms":15943,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a collaborative robot can validate its own cockpit interactions by classifying force, torque, and pose signals with convolutional neural networks, reaching F1-scores of about 95 to 98 percent on most avionics actions.","keywords":["collaborative robotics","deep learning","explainable AI","force-torque sensing","convolutional neural networks","Grad-CAM","avionics testing","success/failure classification"],"falsifier":"Take the trained 1D-CNN models and evaluate them on force, torque, and pose traces collected from the same actions on a real A320 cockpit rather than the simulator, without any retraining or preprocessing changes; a large F1 drop (for example, below 80%) would refute the claim that these models validate real avionics actions, while also revealing how much the result depends on simulator fidelity.","tokens_in":8200,"feed_emoji":"✈️","tokens_out":6602,"duration_ms":58366,"temperature":0.7,"pith_summary":"This paper tries to show that the forces a collaborative robot feels while pressing buttons, flipping switches, and moving levers in an aircraft cockpit are enough to tell, on their own, whether each action succeeded. The authors record force, torque, and tool pose during over 4,330 manually labeled actions on an A320 simulator, preprocess the signals to remove sensor noise and controller oscillations, and train one-dimensional convolutional neural networks to classify each action as success or failure (or as a specific state transition). On five of the six tested components the networks reach F1-scores near or above 97 percent, with a knob-rotation task reaching about 95 percent after dataset augmentation; two-dimensional wavelet and hybrid variants perform slightly lower. The value of the claim, if true, is that automated certification testing can gain a self-contained, wrench-based check that runs in parallel with vision and is explainable through Grad-CAM heatmaps.","feed_headline":"Cobot force signals validate cockpit tests at ~98% F1","feed_subtitle":"1D CNNs on force, torque, and pose classify buttons, switches, and levers as success or failure, with Grad-CAM showing why.","key_machinery":"The load-bearing mechanism is a multi-branch 1D-CNN whose inputs are cleaned by a three-stage preprocessing pipeline: (1) force/torque vectors are rotated from the robot base frame to the tool center point frame via $R^\\top$; (2) a 30 Hz Butterworth filter suppresses sensor noise; and (3) a multi-energy-threshold transient detector picks the most informative 800-sample window of each signal, with thresholds at $0.38$, $0.22$, and $0.12$ of local maximum energy. Highly correlated signals are grouped into dedicated branches with customized kernel sizes and strides; GlobalMaxPooling follows the final convolutional layer, and dense layers map features to success/failure classes. For interpretability, 1D Grad-CAM computes gradient-weighted activations from the last convolutional layer, producing a temporal heatmap that shows which segments of force, torque, or pose drove the prediction. This whole assembly is what the paper argues turns a low-cost force sensor into a reliable validation instrument.","core_discovery":"On the paper's own terms, the central discovery is that 1D convolutional neural networks, trained on preprocessed force, torque, and pose signals, classify cobot actions on cockpit components with near-optimal F1-scores: 97.8% for buttons, 97.4% for switches, 97.5% for FLAP, 98.5% for landing gear, 98.0% for speed brake, and about 95% for knob rotation after addressing class imbalance. The preprocessing pipeline—transforming forces to the tool frame, a 30 Hz Butterworth filter, and an energy-based transient-isolation algorithm that selects an 800-sample window—is what makes the noisy, low-resolution embedded force sensor usable. Grad-CAM, applied in one dimension to preserve temporal structure, shows that the models focus on physically meaningful signal regions (for example, the final force spike of a landing-gear lever that indicates the lever was fully moved). The authors state these findings confirm the feasibility of the proposed wrench-based validation approach.","pith_inferences":["The heuristic threshold values in the transient detector ($0.38$, $0.22$, $0.12$) are a likely source of setup-specific overfitting; a natural extension would be an adaptive or learned segmentation that tunes these thresholds per action type, and testing that would quantify how much of the reported F1 depends on them.","The same pipeline could be applied to other verification tasks where success is a discrete mechanical state change—automotive switchgear, industrial panel operation, medical device controls—by treating the force/torque trace as the ground-truth signature of the state transition.","The paper does not report cross-simulator or cross-platform transfer; a concrete test would be training on the A320 simulator and evaluating on a different cockpit simulator or a real cockpit to measure the domain gap in F1.","Because the knob task only reached its ~95% F1 after augmentation that may have induced data snooping, the paper itself flags this as the weakest result; a cleaner evaluation would collect more natural failures rather than augmenting the minority class."],"forward_implications":["If the central claim holds, automated cockpit certification tests can be validated by the cobot's own force feedback, adding a redundant check alongside computer vision without extra hardware.","The 1D-CNN architecture becomes the recommended choice for this class of wrench-based validation: it matches or beats wavelet and hybrid models while training in under a minute on a consumer GPU.","The preprocessing steps (frame rotation, filtering, transient isolation) can be reused as a template for other contact-rich manipulation tasks where embedded force sensors are noisy.","Since the models are per-action and per-robot, scaling to other cockpit types or robot arms will require collecting new data; the paper's explicit next step is gathering data from diverse simulators and real cockpits.","Explainability via 1D Grad-CAM gives engineers a diagnostic tool: when a failure classification occurs, the heatmap shows which part of the signal triggered it, facilitating root-cause analysis."],"supporting_citations":[{"why":"Supplies the precedent of using CNNs on force signals for contact-rich robotics, inspiring the DL architectures developed in this research.","marker":"[9]"},{"why":"Supplies the energy-based transient signal detection method that the preprocessing pipeline adapts to isolate the informative segment of each force/torque trace.","marker":"[11]"},{"why":"Supplies the continuous wavelet transform reference (Morlet wavelet) used to build the 2D scaleograms for the 2D and hybrid CNN models.","marker":"[14]"},{"why":"Supplies the hybrid-CNN architecture combining 1D and 2D representations, which the paper adapts into its multi-branch hybrid models.","marker":"[17]"},{"why":"Supplies the class activation mapping concept that underpins the paper's Grad-CAM approach for visual explanations.","marker":"[18]"},{"why":"Supplies the 1D Grad-CAM variant used to produce temporal heatmaps on time-series force signals, the paper's main interpretability tool.","marker":"[19]"}],"fun_headline_variants":["CNN on cobot forces grades cockpit actions at 98% F1","Force-signal AI verifies aircraft panel operations with 98% F1","Deep learning validates cobot switch flips via force patterns","Cobot force analysis with explainable AI hits 98% F1","Explainable CNN on force signals confirms cockpit actions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The manual labels assigned to the recorded actions are treated as correct ground truth for success or failure, and the A320 simulator's force response is assumed to represent real aircraft cockpit components, so if either is wrong the reported F1-scores do not measure true validation accuracy.","fun_headline_variants_meta":{"raw":{"variants":["CNN on cobot forces grades cockpit actions at 98% F1","Force-signal AI verifies aircraft panel operations with 98% F1","Deep learning validates cobot switch flips via force patterns","Cobot force analysis with explainable AI hits 98% F1","Explainable CNN on force signals confirms cockpit actions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000552,"raw_usage":{"total_tokens":2621,"prompt_tokens":926,"completion_tokens":1695,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":1606}},"tokens_in":542,"tokens_out":1695,"duration_ms":12619,"temperature":1.0,"reasoning_tokens":1606,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:13:32.135236+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained 1D-CNN models and evaluate them on force, torque, and pose traces collected from the same actions on a real A320 cockpit rather than the simulator, without any retraining or preprocessing changes; a large F1 drop (for example, below 80%) would refute the claim that these models validate real avionics actions, while also revealing how much the result depends on simulator fidelity.","supporting_citations":[{"cited_title":"https://doi.org/10.1016/j.proeng.2012.01.843","cited_arxiv_id":null,"evidence_quote":"Supplies the energy-based transient signal detection method that the preprocessing pipeline adapts to isolate the informative segment of each force/torque trace."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the hybrid-CNN architecture combining 1D and 2D representations, which the paper adapts into its multi-branch hybrid models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 1D Grad-CAM variant used to produce temporal heatmaps on time-series force signals, the paper's main interpretability tool."}],"review_version":1}