{"id":"9be35478-ca72-417d-a494-f32c6b9b3cc2","arxiv_id":"1908.07319","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A fully convolutional network with class activation maps reaches competitive accuracy for surgical skill classification and OSATS regression on JIGSAWS, while localizing motion segments that drive each prediction.","lead":"A computer model learns to classify surgical skill and predict surgical quality scores from the motion of robot tools during simulated surgery, and it highlights which motion segments drove each decision. The approach promises objective, interpretable feedback for surgical trainees, but the results rest on a very small dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LOSO leaves the same surgeon's other trials in training, so the reported 100% accuracy may reflect subject identity rather than surgical skill; the paper's Section 5 limitation confirms this.","rationale":"The reader's weakest assumption correctly identifies the identity-versus-skill confound as the central risk and recommends a conditional verdict. My read strengthens this concern by locating the exact protocol: Section 3's LOSO is leave-one-super-trial-out, not leave-one-user-out, so the held-out trial's subject still has other trials in training. This makes the identity shortcut even easier than the reader's 'held-out subject' framing suggests and is explicitly acknowledged in Section 5 ('a surgeon should be present in the training set'). The proposed LOUO re-run directly tests whether the reported accuracy survives surgeon-independent evaluation; if it does not, the central interpretable-skill-evaluation claim is unsupported. Because this is a falsifiable, fixable condition rather than a demonstrated failure of the method on a valid protocol, the reader's CONDITIONAL verdict remains appropriate. I do not see an independent flaw in the CAM derivation (Eq. 2) or the FCN architecture, and the authors' release of source code is a positive. The main missing evidence is user-independent accuracy and per-subject variance, which the Section 5 limitation already flags.","tokens_in":9372,"tokens_out":12353,"duration_ms":126538,"concrete_test":"Re-run the classification experiments in Table 1 with leave-one-user-out cross-validation: for each of the 8 surgeons, train on all trials of the other 7 surgeons and test on all 5 trials of the held-out surgeon, then report per-surgeon micro/macro accuracy and a skill-level confusion matrix. If the mean held-out-surgeon accuracy drops substantially from the reported 100%/92.1% (for example toward chance or below roughly 75%), the original LOSO numbers are inflated by within-subject identity leakage. As a secondary check, train a classifier to predict which of the 8 subjects produced each kinematic trial; near-perfect subject-identity accuracy would confirm that the kinematic traces carry strong subject-specific signatures that the FCN can exploit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the FCN evaluates surgical skill and reaches state-of-the-art accuracy on JIGSAWS. The load-bearing weakness is the evaluation protocol. Section 3 states: 'We adopted for both classification and regression tasks a leave-one-super-trial-out (LOSO) scheme [1].' In JIGSAWS, a super trial is one trial from each of the eight subjects; doing LOSO therefore holds out one trial per subject while retaining that same subject's other four trials in the training set. Section 5 concedes the consequence: 'a surgeon should be present in the training set in order to make a prediction.' Under this protocol, a network can classify a held-out trial by recognizing the subject's individual motor signature from his or her remaining training trials. The 100% accuracy for suturing and needle passing in Table 1 is then evidence of within-subject identity recognition, not of a generalizable skill assessment. The Section 5 remark that only two experts exist reinforces the point: even a proper leave-one-user-out split would have only one expert per fold, so the dataset cannot independently support the expert-generalization claim. The reader's 'identity vs skill' concern is correct, but the mechanism is stronger than classifying an unseen subject: the exact test subject is present in training.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a fully convolutional neural network (FCN) for surgical skill evaluation from kinematic data in the JIGSAWS dataset. The architecture applies grouped convolutions over the 76 kinematic channels (organized by manipulator and kinematic type), global average pooling, and class activation maps to produce both skill classifications (novice/intermediate/expert) and OSATS regression scores. Three separate models are trained for suturing, needle passing, and knot tying. Results are reported under a leave-one-super-trial-out (LOSO) scheme and compared with published S-HMM, ApEn, SAX-VSM, and CNN baselines. The authors report 100% micro/macro accuracy for suturing and needle passing, 92.1%/93.2% for knot tying, and Spearman correlations of 0.60, 0.57, and 0.65 for OSATS regression, and they claim state-of-the-art performance as well as interpretable feedback via CAM heatmaps.","tokens_in":9639,"tokens_out":8566,"duration_ms":85392,"significance":"If the reported results were valid evidence of skill-level generalization, the paper would make a useful practical contribution: it offers an end-to-end architecture that avoids manual gesture segmentation, provides per-trial heatmaps via CAM, and ships code and comparisons to existing baselines. The grouping of kinematic channels by manipulator and physical meaning is a reasonable inductive bias. However, the current evidence does not support the broad claim of accurate and interpretable surgical-skill evaluation: the LOSO protocol leaves the same subject's other trials in the training set, and the interpretability component is not validated. The significance is therefore conditional on substantially stronger evaluation or on appropriately narrowed claims.","major_comments":[{"comment":"The evaluation protocol cannot distinguish skill-level generalization from subject-identity recognition. Under leave-one-super-trial-out, the held-out trial comes from a subject whose other trials are in the training set, so the network can classify by matching an individual motor signature rather than by any skill-level concept. The paper's own Section 5 states that 'a surgeon should be present in the training set in order to make a prediction' and that this is 'a huge problem originating from the limited dataset size.' This is not a peripheral caveat: the 100% accuracies in Table 1 are the main evidence for the paper's central claim. The authors should report a leave-one-user-out experiment, a subject-identification control, or per-subject confusion matrices, and must temper the state-of-the-art claim according to what those results show.","section":"Section 3 (LOSO) and Table 1"},{"comment":"The reported FCN accuracies are point estimates without dispersion. The text states that results are averaged over 40 runs, but no standard deviation, confidence interval, or significance test is given. With only 40 trials per task (8 subjects x 5 trials), one misclassification changes micro accuracy by 2.5 percentage points, so the 92.1% versus 97.4% difference for knot tying is within the sampling error of a handful of trials. The claim that the FCN is 'much more accurate' than S-HMM is therefore unsupported; per-fold or per-run results, or confidence intervals, are required.","section":"Section 4 and Table 1"},{"comment":"Spearman's rho is insufficient to support the regression claim. The paper reports only rank correlation averaged over the six OSATS targets, without absolute error, calibration, per-target breakdown, or uncertainty. The values (0.60, 0.57, 0.65) are modest, and the comparison is made against only one published method (ApEn). Because the same LOSO protocol applies, the same subject-identity confound affects the regression results. The authors should report mean absolute error or RMSE, per-target correlations with confidence intervals, and ideally subject-independent performance.","section":"Section 4 (regression)"},{"comment":"The interpretability claim is not validated. A CAM heatmap is by construction a linear combination of the same weights that produced the prediction; it shows which parts of the input the model used, not that those parts correspond to meaningful surgical gestures or useful feedback. The text interprets the highlighted subsequence as the gesture 'pulling suture' without any gesture annotation or expert verification. Section 5 itself concedes that the feedback technique would benefit from real use-case validation. The conclusion should be limited to 'the model highlights discriminative segments' unless the authors provide evidence that the highlighted segments are clinically meaningful.","section":"Section 4 and Figures 2-3"}],"minor_comments":[{"comment":"The term 'super trial' is used in the LOSO description but never defined in this paper; the reader should be told that a super trial contains one trial from each of the eight subjects.","section":"Section 3"},{"comment":"Section 3 says the effects of the learning rate and l2 regularization are discussed in Section 4, but Section 4 contains no such sensitivity analysis; either add the analysis or remove the pointer.","section":"Section 3 and Section 4"},{"comment":"The macro metric is described as 'boiling down to computing the precision metric'; the exact formula should be given, since macro-averaged precision and macro-averaged recall can differ.","section":"Section 4 (macro measure)"},{"comment":"The explanation for the lower knot-tying accuracy (mean OSATS 17.7 versus 17.1 for experts and intermediates) is asserted without supporting data; report the class distributions and score distributions for that task.","section":"Section 4 (knot tying)"},{"comment":"The statement that the proposed method does 'not sacrifice accuracy for interpretability' is a comparison to the SAX-VSM baseline only; no controlled comparison is made against an equally accurate but less interpretable model.","section":"Section 4 (interpretability comparison)"},{"comment":"The abstract says 'new competitive state-of-the-art performance,' while the conclusion says 'new state-of-the-art performance'; Table 1 shows the proposed method is not the best on the knot-tying regression task, so the wording should be consistent and precise.","section":"Abstract and Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The authors are transparent in Section 5 about the dataset's limitations, which is to their credit; however, the abstract and conclusion are not scoped accordingly. The LOSO protocol is the standard JIGSAWS benchmark, so the subject-identity confound is partly a field-level limitation, but the paper should at minimum separate the benchmark claim from the skill-generalization claim. I do not see a novelty disclosure issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a clean, honest application of fully convolutional networks plus class activation maps to surgical skill evaluation on JIGSAWS. The architecture is sensible, with domain-informed channel grouping that is easy to follow. The authors include code, compare against several published baselines, and explicitly flag the dataset's limitations in their Section 5. What's genuinely new here is the regression extension that predicts six OSATS subscores, and the demonstration that CAM heatmaps can be produced for both classification and regression. The classification numbers themselves already appeared in their MICCAI 2018 paper, so the incremental novelty is modest.\n\nThe soft spot is load-bearing. They use leave-one-super-trial-out, which holds out one trial per subject while leaving that same subject's other four trials in the training set. As Section 5 admits, 'a surgeon should be present in the training set in order to make a prediction.' Under that protocol, a network can identify the subject's individual motor signature and classify held-out trials by identity rather than by skill. The 100% accuracy on suturing and needle passing is thus far less impressive than it looks—it may say nearly nothing about generalizing to an unseen surgeon. The stress-test note is correct, and the Section 5 concession effectively confirms it.\n\nOther issues are real but secondary. The dataset has only eight subjects, two of them experts; Table 1 gives point estimates without variance despite averaging over 40 runs; and the regression evaluation uses only rank correlation, not calibration or error magnitude. The CAM analysis is illustrative, but the authors themselves say it needs validation with expert surgeons. On the positive side, the paper does not oversell: the limitations section is unusually candid, and the comparison with prior work is fair.\n\nWho gets value from this? Someone working on surgical data science who wants a clear example of an interpretable deep model on JIGSAWS and a cautionary tale about evaluation protocols. It is not a paper whose headline accuracy should be cited as evidence of skill classification performance.\n\nFor peer review: yes, it deserves a serious referee—but the referee should push for a proper leave-one-user-out analysis (with the caveat that only two experts make that near-impossible on this dataset), error bars, and a direct test of identity versus skill. Without such a revision, the central claim doesn't hold. As it stands, it's a useful methodological prototype, not a validated result.","headline":"Competent application of known deep-learning components to a small benchmark, but the reported 100% accuracies are very likely inflated by a leave-one-trial-out protocol that keeps each test subject's other trials in training.","tokens_in":10163,"tokens_out":1038,"would_cite":false,"duration_ms":13725,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fully convolutional network can classify surgical skill from raw kinematics at 100% accuracy on two of three tasks, and can point to the movements behind each decision.","keywords":["surgical skill assessment","kinematic data","fully convolutional networks","global average pooling","class activation map","interpretable machine learning","time series classification","JIGSAWS"],"falsifier":"Train the same network to say which of the eight surgeons is moving, using the same one-surgeon-out testing scheme; if it identifies held-out surgeons from their motion about as well as or better than it grades their skill, the skill results could be driven by identity rather than expertise.","tokens_in":9181,"feed_emoji":"🤖","tokens_out":13754,"duration_ms":125460,"temperature":0.7,"pith_summary":"Manual assessment of surgical trainees by expert observers is slow, expensive, and subjective. This paper argues that a fully convolutional network trained on raw kinematic recordings of robotic surgery can evaluate skill automatically and, unlike earlier black-box classifiers, can explain its decisions. On the JIGSAWS dataset the reported accuracy is 100% micro and macro for suturing and needle passing, and 92.1% micro and 93.2% macro for knot tying, with OSATS score predictions comparable to a published entropy-based baseline. Because the network processes an entire trial without manual gesture boundaries, it points toward routine, objective, interpretable feedback for trainees.","feed_headline":"Surgical skill graded from motion alone, with a heatmap of why","feed_subtitle":"Neural network reaches 100% accuracy on two of three surgical tasks and pinpoints the movements behind each score.","key_machinery":"The load-bearing component is the fully convolutional network whose last convolutional feature map is passed through global average pooling instead of a dense layer. Because the pooled feature for filter $k$, $\\sum_t \\mathcal{A}_k(t)$, is weighted by $w^c_k$, the class activation map $\\mathcal{M}_c(t)=\\sum_k w^c_k \\mathcal{A}_k(t)$ decomposes the output score into per-time-step contributions. This identity is what turns the classifier into an explainer: after training, plotting $\\mathcal{M}_c(t)$ over the trial's trajectory shows which kinematic subsequences were responsible for classifying a surgeon as novice, intermediate, or expert, or for predicting an OSATS sub-score. The convolutions are also structured by domain knowledge, with separate filters for each manipulator and for kinematic subgroups (position, linear and rotational velocity, rotation matrix, gripper speed), and the use of global pooling lets the network accept trials of variable length.","core_discovery":"The paper's central claim is that an end-to-end fully convolutional network with global average pooling can both grade and explain surgical skill from the 76 kinematic channels recorded during robot-assisted surgery. For classification, the network outputs novice, intermediate, or expert; for regression, it predicts each of the six modified OSATS sub-scores. Under leave-one-super-trial-out validation on JIGSAWS, the authors report 100% micro and macro accuracy on suturing and needle passing, 92.1% micro and 93.2% macro on knot tying, and mean rank correlations of 0.60, 0.57, and 0.65 across the three tasks. The explanation comes from a class activation map, derived from the global average pooling weights, which assigns a contribution value to each time step and highlights which motion segments drove the prediction.","pith_inferences":["One implication left implicit is that the same global-pooling/class-activation machinery applies to other multivariate time-series tasks where practitioners need to know why a prediction was made, such as rehabilitation monitoring; the per-time-step attribution formula carries over unchanged.","A useful control the paper does not report is an identity-classification experiment: if an eight-way 'which surgeon is moving' classifier reaches similar one-surgeon-out accuracy, part of the skill signal could be subject identity rather than skill level.","The hand-built channel grouping is a design choice; ablating it by feeding all 76 channels through ungrouped convolutions would quantify how much of the accuracy comes from the kinematic-domain structure versus the fully convolutional network itself."],"forward_implications":["Skill evaluation could run automatically after each training trial, without an expert surgeon in the loop and without manual gesture segmentation.","The heatmaps give trainees concrete, segment-level feedback, such as which 'pulling suture' motion marked them as novice, rather than just a single numeric score.","Because the architecture accepts variable-length trials, no resampling or window-based preprocessing is needed to compare performances across trials.","On the JIGSAWS benchmark, the method matches or beats the published baselines: 100% on suturing and needle passing, 92.1%/93.2% on knot tying, and rank correlations comparable to or higher than the entropy-based approach on all tasks."],"supporting_citations":[{"why":"This reference supplies the kinematic trials, skill labels, OSATS scores, and the overall evaluation setting used throughout the paper.","marker":"[6]"},{"why":"This reference defines the micro and macro metrics and the leave-one-super-trial-out validation protocol used for all comparisons.","marker":"[1]"},{"why":"This reference is the gesture-dependent sparse hidden Markov model baseline that the proposed method outperforms without manual gesture segmentation.","marker":"[19]"},{"why":"This reference is the approximate-entropy baseline whose classification accuracy and rank correlation values are the main quantitative comparison.","marker":"[24]"},{"why":"This reference is the interpretable bag-of-words baseline that the paper says is less accurate, establishing the accuracy-interpretability tradeoff the FCN claims to avoid.","marker":"[4]"},{"why":"This reference is the earlier CNN baseline that depends on input length, which the proposed architecture improves on by using global average pooling.","marker":"[21]"},{"why":"This reference supplies the class activation map technique that converts global-pooling weights into per-time-step attributions.","marker":"[23]"},{"why":"This reference supplies the fully convolutional time-series architecture and the extension of class activation maps to time series on which the method builds.","marker":"[22]"},{"why":"This reference is the authors' earlier FCN classifier for surgical skill, extended here to OSATS regression and interpretable feedback.","marker":"[11]"}],"fun_headline_variants":["AI grades surgical motion and highlights the why","Surgical skill from motion, with built-in explanations","CNN scores surgeons and maps which moves matter","No more black box: AI explains surgery skill grades","From motion to mastery: CNN shows what makes an expert"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that testing on each surgeon one at a time measures surgical skill rather than which surgeon is moving; with only two experts in the dataset, the model might recognize an individual's motion patterns instead of genuine expertise.","fun_headline_variants_meta":{"raw":{"variants":["AI grades surgical motion and highlights the why","Surgical skill from motion, with built-in explanations","CNN scores surgeons and maps which moves matter","No more black box: AI explains surgery skill grades","From motion to mastery: CNN shows what makes an expert"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000397,"raw_usage":{"total_tokens":2078,"prompt_tokens":945,"completion_tokens":1133,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":1060}},"tokens_in":561,"tokens_out":1133,"duration_ms":12230,"temperature":1.0,"reasoning_tokens":1060,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:20:22.612252+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same network to say which of the eight surgeons is moving, using the same one-surgeon-out testing scheme; if it identifies held-out surgeons from their motion about as well as or better than it grades their skill, the skill results could be driven by identity rather than expertise.","supporting_citations":[{"cited_title":"In: Modeling and Monitoring of Computer Assisted In- terventions MICCAI Workshop","cited_arxiv_id":null,"evidence_quote":"This reference supplies the kinematic trials, skill labels, OSATS scores, and the overall evaluation setting used throughout the paper."},{"cited_title":"IEEE Transactions on Biomedical Engineering 64(9):2025–2041","cited_arxiv_id":null,"evidence_quote":"This reference defines the micro and macro metrics and the leave-one-super-trial-out validation protocol used for all comparisons."},{"cited_title":"In: Information Pro- cessing in Computer-Assisted Interventions, pp 167–177","cited_arxiv_id":null,"evidence_quote":"This reference is the gesture-dependent sparse hidden Markov model baseline that the proposed method outperforms without manual gesture segmentation."},{"cited_title":"International Journal of Computer As- sisted Radiology and Surgery 13(5):731–739","cited_arxiv_id":null,"evidence_quote":"This reference is the approximate-entropy baseline whose classification accuracy and rank correlation values are the main quantitative comparison."},{"cited_title":"In: Artificial Intelli- gence in Medicine, pp 136–145","cited_arxiv_id":null,"evidence_quote":"This reference is the interpretable bag-of-words baseline that the paper says is less accurate, establishing the accuracy-interpretability tradeoff the FCN claims to avoid."},{"cited_title":"International Journal of Com- puter Assisted Radiology and Surgery 13(12):1959–1970","cited_arxiv_id":null,"evidence_quote":"This reference is the earlier CNN baseline that depends on input length, which the proposed architecture improves on by using global average pooling."},{"cited_title":"In: IEEE Conference on Computer Vision and Pattern Recognition, pp 2921–2929","cited_arxiv_id":null,"evidence_quote":"This reference supplies the class activation map technique that converts global-pooling weights into per-time-step attributions."},{"cited_title":"In: International Joint Conference on Neural Net- works, pp 1578–1585","cited_arxiv_id":null,"evidence_quote":"This reference supplies the fully convolutional time-series architecture and the extension of class activation maps to time series on which the method builds."},{"cited_title":"In: Inter- national Conference On Medical Image Computing and Computer Assisted Intervention, pp 214–221","cited_arxiv_id":null,"evidence_quote":"This reference is the authors' earlier FCN classifier for surgical skill, extended here to OSATS regression and interpretable feedback."}],"review_version":1}