{"id":"296e642a-0fd7-4991-89cc-268e2228efac","arxiv_id":"2412.15819","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A GAN discriminator trained on CNN features improves rejection of unknown gestures in EMG control, but only within the same subject, and the threshold is tuned on real unknown samples.","lead":"EMG-based prosthetics often misclassify unfamiliar gestures, which can cause unintended movements. This paper reports that a GAN-based filter that rejects 'unknown' gestures lowers one error metric in within-subject tests, but cross-subject performance stays poor.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation protocol leaks real unknown labels into discriminator training and threshold selection, so the reported 23.6% AER improvement may not reflect true open-set generalization.","rationale":"The reader's weakest assumption was that the synthetic unknown distribution must be a sufficient proxy for real unknowns. That is a real concern, but the more load-bearing problem is the evaluation protocol itself: the paper's own text indicates that real unknown samples are used to select the discriminator threshold (Fig. 6, Section V.B) and possibly to train/select the discriminator (Section VI.A). This directly undermines the central claim because the discriminator is not being tested on truly unseen classes. Even if the threshold is only tuned on a validation split, the reported AER improvement is not an unbiased estimate of deployment performance, where no unknown labels are available. The cross-domain experiments with AER above 80% strongly suggest that the method does not transfer to new subjects or unknown classes once the threshold is not tuned on that domain's unknowns. The internal contradiction between Section IV.C and Section VI.A needs to be resolved. I recommend conditional acceptance because the method may still be valid after a rigorous re-evaluation with properly held-out unknowns and a fixed threshold, but the current evidence does not support the headline claim as stated. The reader's emphasis on synthetic-versus-real distribution mismatch is related but secondary; the threshold tuning issue is more fundamental and should be addressed first.","tokens_in":13832,"tokens_out":4135,"duration_ms":36586,"concrete_test":"Obtain the training code (the GitHub link in the paper is broken) and check whether features from real unknown gesture classes are used in GAN training or in ROC threshold selection. Then rerun the self-collected experiment with a strict protocol: train the discriminator using only known-class features and synthetic unknowns; fix the threshold using only a known-class validation set; and finally evaluate AER on held-out real unknowns. If the AER improvement falls well below 23.6%, the reported improvement is an artifact of label leakage in threshold selection or training.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that a GAN-based discriminator can reject previously unseen gestures. For this claim to hold, the discriminator must not see real unknown samples during training, and the rejection threshold must not be chosen using real unknown labels. The paper contradicts itself on both points. Section IV.C states that unknown EMG samples are preserved for final evaluation and that the generator synthesizes unknown samples for discriminator training. However, Section VI.A says that classifiers predict features from 'all known and unknown class data' and that these features are used 'to create a new dataset for training the GAN and selecting the discriminator.' This means real unknown samples enter GAN training or at least discriminator selection in the ratio experiments. Additionally, Section V.B and Fig. 6 explicitly state that the ROC curve and optimal cutoff threshold are determined using the test set derived from self-collected data, which contains the real unknown gestures used in Section VI.D. Thus the 97.6% known-class accuracy and the 23.6% AER improvement are measured with a threshold chosen using labels of the very unknown samples being rejected. The threshold, and possibly the discriminator itself, is therefore overfit to the specific unknown gestures in the evaluation set. This makes the headline result an artifact of the evaluation protocol rather than evidence of open-set generalization. The poor cross-domain results (AER above 80% in Section VI.C) are consistent with this interpretation: the discriminator does not generalize to new unknown classes or subjects once the tuned threshold no longer matches. The paper's self-reported limitation in cross-domain scenarios further supports that the within-session success is driven by distributional similarity to the validation unknowns, not by a robust open-set mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a GAN-based open-set recognition framework for myoelectric control. A CNN classifier first maps EMG windows into class probabilities for known gestures; a GAN discriminator is then trained on known-class features together with synthetic features produced by a generator, and a threshold on the discriminator output is used to reject presumed unknown gestures before an actuator executes a movement. The method is evaluated on the Ninapro DB1 dataset and on self-collected surface-EMG data from ten gestures, reporting 97.6% accuracy on known classes and a 23.6 percentage-point improvement in Active Error Rate after rejection. Additional experiments vary the known-to-unknown ratio, test new unknown classes, and probe cross-domain (cross-subject) performance.","tokens_in":14060,"tokens_out":3148,"duration_ms":26142,"significance":"If the evaluation protocol were sound, this paper would make a useful and practically relevant contribution: it tackles the underexplored problem of unknown gestures in myoelectric control, the proposed module is lightweight and suitable for edge deployment, and the authors provide publicly available code and a real-hardware validation with a Shimmer device. These are genuine strengths. However, the headline results rest on an evaluation protocol in which real unknown samples are used to select the rejection threshold (Sec. IV.C, Fig. 6) and, in the ratio experiments, to create the feature set for GAN training and discriminator selection (Sec. VI.A). This makes the reported 97.6% accuracy and 23.6% AER improvement artifacts of tuning to the target unknown distribution rather than evidence of open-set generalization. The cross-domain experiments (Sec. VI.C, Fig. 11) show AER above 80% for all methods, further limiting the claims to within-session conditions. The central idea is worth pursuing, but the current experiments need to be redesigned before the claims can be accepted.","major_comments":[{"comment":"The rejection threshold is selected as the point closest to the upper-left corner of the ROC curve computed on the test set derived from self-collected data, and this test set contains the exact real unknown gestures later used in Sec. VI.D to report the 97.6% known-class accuracy and the 23.6% AER improvement. Because the threshold is tuned using labels of the very unknown samples being rejected, the headline result measures threshold overfitting to the evaluation set rather than open-set generalization. The threshold should be chosen using only known-class data, synthetic unknowns, or a separate validation set disjoint from the unknown classes used for final evaluation.","section":"Section IV.C, Fig. 6"},{"comment":"The text states that classifiers predict features from 'all known and unknown class data' and that these features are used 'to create a new dataset for training the GAN and selecting the discriminator.' This directly contradicts Sec. IV.C, which says the unknown EMG samples are preserved for the final evaluation of the discriminator. In the ratio experiments, real unknown samples therefore enter GAN training or discriminator selection, so the AUC values in Table I, the F1-scores in Table II, and the AER comparisons in Figs. 8-10 do not measure rejection of previously unseen gestures. The authors need to specify exactly which real unknown samples are used for training versus evaluation and re-run the experiments with strict separation.","section":"Section VI.A"},{"comment":"In cross-domain testing, the AER values exceed 80% for all methods, including the proposed OpenGAN approach. This indicates that the method's apparent success is confined to the same subject and recording session, and it does not support the abstract's general claim of robustness under 'inter-subject/session variability.' The conclusions should be explicitly scoped to the within-session setting, or the cross-domain performance must be improved before the claimed real-world applicability is justified.","section":"Section VI.C, Fig. 11"}],"minor_comments":[{"comment":"The phrase 'spited into known and unknown dataset' appears to be a typo for 'split into known and unknown datasets.'","section":"Section IV.A"},{"comment":"The caption contains the typo 'inidcates' instead of 'indicates.'","section":"Fig. 9 caption"},{"comment":"The caption refers to 'the right heatmap plot' and 'the right error plot,' but the text describes what appears to be a left heatmap and a right error plot; the figure-caption orientation should be corrected.","section":"Fig. 13 caption"},{"comment":"The phrase '57.6% open set accuracy' is unclear; it should be defined explicitly as the accuracy on the combined known-plus-unknown test set before rejection, so that the improvement to 81.2% is unambiguous.","section":"Section VI.D"},{"comment":"The generator's hidden dimension NHidden is referenced in the figure and in Sec. IV.C, but no numerical value is reported in the experiments; please state the value used for each evaluation.","section":"Fig. 5"},{"comment":"The notation is inconsistent between the introduction's 'K-patterns' and the later 'Nknown'; a single notation for the number of known classes would improve readability.","section":"Section IV.B"}],"recommendation":"major_revision","confidential_remarks":"The core idea is plausible and the topic fits the journal, but the evaluation protocol leaks real unknown labels into threshold selection and, in the ratio experiments, into discriminator training. The authors should be asked to re-run all experiments with a strict split that keeps the final unknown classes entirely out of threshold selection and GAN training, and to report whether the improvements survive under that protocol. If the results do not survive, the paper should be rejected; if they do, a revised version with scoped claims could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: the paper's headline number is probably an artifact of the evaluation protocol. The rejection threshold is chosen from the ROC curve of a test set that contains the real unknown gestures, and Sec. VI.A states that unknown class data are used to create the dataset for training the GAN and selecting the discriminator. That directly contradicts Sec. IV.C, which says unknown samples are preserved for final evaluation. So the 23.6% AER improvement and the 97.6% accuracy are measured with a threshold tuned to the very unknowns being rejected. The cross-domain results (AER above 80% everywhere) are consistent with that: the discriminator does not generalize to new subjects or new unknown classes once the tuned threshold no longer matches.\n\nWhat is genuinely there: the application of a GAN generator/discriminator to CNN output features for EMG open-set rejection is a sensible, lightweight idea, and the AER framing is clinically meaningful. The Ninapro DB1 experiments cover a range of known-to-unknown ratios and show the method beats a plain open classifier in-domain. If the protocol were clean, this would be a useful incremental contribution.\n\nThe soft spots are not minor. The threshold leakage is load-bearing. The paper also does not provide a fixed-threshold cross-subject evaluation; its own cross-domain numbers show near-zero benefit. The code link in the paper is broken, which hurts reproducibility. The novelty is modest: the GAN-based open-set machinery comes straight from Schlegl and Neal, and the contribution is the application to EMG features.\n\nWho gets value: researchers working on myoelectric control robustness could use this as a starting point, but they should not take the headline numbers at face value. The paper deserves a serious referee because the problem is real and the method is plausible; the flaws are fixable with a cleaner protocol and honest reporting.\n\nMy recommendation: send it to peer review with major revision, and require the authors to (1) retrain without any real unknown samples in GAN training or threshold selection, (2) report AER with a threshold fixed at training time, and (3) provide working code. If those fixes land, the paper becomes a solid application note.","headline":"The claimed 23.6% AER improvement is likely a protocol artifact because the rejection threshold is tuned on real unknown samples; the method is a modest, plausible application of known GAN-based open-set ideas to EMG.","tokens_in":14694,"tokens_out":2479,"would_cite":false,"duration_ms":20513,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a GAN-based discriminator operating on CNN output features can identify and reject previously unseen gesture classes in myoelectric control, cutting the active error rate by 23.6% while keeping known-class accuracy…","keywords":["surface electromyography","myoelectric control","open-set recognition","generative adversarial network","gesture recognition","active error rate","unknown gesture rejection","Ninapro DB1"],"falsifier":"Train the CNN and GAN on one subject's known gestures, then test the trained discriminator on real unknown gestures recorded from a different subject or on a different day; if the discriminator's AUC falls to roughly 0.5 or the active error rate stays above 80%, the synthetic-unknown proxy has failed. A sharper test is to withhold all real unknown gestures from threshold selection and set the threshold using only the GAN's synthetic samples; if the rejection performance then collapses to the no-discriminator baseline, the claimed gain depends on validation-time access to real unknowns rather than on the GAN's synthetic distribution.","tokens_in":13609,"feed_emoji":"🦾","tokens_out":4666,"duration_ms":40496,"temperature":0.7,"pith_summary":"These authors are trying to establish that an open-set recognition layer—a small GAN discriminator placed in front of a CNN classifier—can make myoelectric control safer by rejecting gestures the system has never seen, rather than forcing every input into a known category. The reason this matters is that unintended movements are the main barrier to clinical adoption of prosthetic and exoskeleton control; rejecting an unknown motion is cheaper than executing a wrong one. On the Ninapro DB1 dataset and on their own six-channel recordings, they report known-class accuracy of 97.6% and a 23.6% improvement in active error rate after the discriminator rejects unknown gestures. The whole pipeline is light enough to run on edge devices, which is where such safeguards would actually be deployed.","feed_headline":"GAN guard rejects unseen gestures, cutting myoelectric errors 23.6%","feed_subtitle":"A small discriminator in front of the CNN tells familiar motions from unfamiliar ones, blocking wrong prosthetic movements before they…","key_machinery":"The load-bearing object is the GAN discriminator applied to the CNN's output feature vector, a $1 \\times N_{\\text{known}}$ probability vector. The generator takes Gaussian noise and emits synthetic feature vectors meant to resemble what an unknown gesture would produce; the discriminator is trained with cross-entropy loss to output high scores for real known-gesture features and low scores for synthetic unknowns, and a threshold is fixed from the ROC curve to convert scores into accept or reject decisions. This design moves the burden of open-set recognition off the raw EMG signal and onto a low-dimensional feature space, which keeps the added computation small enough for edge deployment.","core_discovery":"The paper's central claim is that a discriminator trained adversarially on the output features of a CNN gesture classifier can separate known gestures from unknown ones well enough to block the unknown ones before they reach the actuator. The discriminator sees only the K-dimensional prediction vector of a CNN trained on known gestures; the generator manufactures synthetic 'unknown' prediction vectors from Gaussian noise, and the discriminator learns to tell the CNN's real known-gesture outputs from these synthetic unknowns. At run time, if the discriminator's score for a new sample falls below a threshold chosen from the ROC curve, the system holds its default or last state instead of moving. Within the same subject and recording session this reduces the proportion of executed actions that are wrong by 23.6% while preserving 97.6% accuracy on known classes; the paper shows the improvement is consistent across several known-to-unknown ratios, though it degrades sharply in cross-subject tests.","pith_inferences":["A deployed system would need a way to set the rejection threshold without access to real unknown gestures, since the paper's threshold is chosen from an ROC curve built with validation unknowns; how to set it from synthetic data alone is left open.","The same discriminator-on-features design could be attached to any closed-set classifier that outputs a fixed-size prediction vector, so the idea may transfer to other biosignal control loops such as EEG or EOG-based interfaces.","Combining this unknown-gesture guard with confidence-based rejection for known gestures would address two separate error sources; the paper names this as future work, and the experimental trend suggests the two mechanisms are complementary.","Because the generator only sees CNN output vectors, the approach is agnostic to the underlying classifier architecture; this means stronger or weaker classifiers can be swapped in without changing the open-set rejection layer."],"forward_implications":["Deploying this guard means an EMG-driven prosthetic or exoskeleton will stay still rather than perform an unintended motion when it encounters an unfamiliar gesture, improving clinical usability.","The approach works across known-to-unknown ratios up to about 1:2; beyond that, with many more unknown than known gestures, AER grows and the method loses its advantage.","Because the discriminator operates on the CNN's prediction vector rather than on raw signals, it can be added to an already-trained gesture classifier without retraining the classifier.","The 97.6% known-class accuracy and 23.6% AER improvement were measured within a single recording session; the paper's cross-subject tests show AER above 80%, so within-session deployment is the realistic near-term target."],"supporting_citations":[{"why":"Supplies the Ninapro DB1 public dataset used for the main ratio and cross-domain experiments.","marker":"[25]"},{"why":"Introduces the GAN formulation whose generator and discriminator losses the paper adapts.","marker":"[24]"},{"why":"Establishes the idea of using a GAN generator to produce synthetic outlier data for training an open-set discriminator.","marker":"[23]"},{"why":"Motivates active error rate as the usability metric by arguing that halting a wrong action costs less than executing it.","marker":"[15]"},{"why":"Defines confidence-based rejection after LDA and provides the performance-measure framework the paper extends to unknown gestures.","marker":"[17]"},{"why":"Provides the earlier selective-classification baseline that rejects low-confidence outputs to reduce AER.","marker":"[16]"},{"why":"Documents the inter-subject/session variability in EMG recognition (about 66% accuracy across repetitions) that motivates the robustness problem.","marker":"[6]"}],"fun_headline_variants":["GAN discriminator blocks unknown gestures, slashing myoelectric errors 23.6%","Open-set GAN rejects unseen gestures, improving myoelectric control by 23.6%","Unknown gesture? GAN says no, cutting myoelectric errors 23.6%","Myoelectric control gets GAN guard, 23.6% fewer errors","GAN guard for myoelectric control cuts errors 23.6%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that synthetic unknown-gesture features generated by a GAN stand in for real unknown-gesture features well enough that a threshold tuned on real validation unknowns will also reject real unknowns at run time, and the evidence suggests this holds within one recording session but not across subjects.","fun_headline_variants_meta":{"raw":{"variants":["GAN discriminator blocks unknown gestures, slashing myoelectric errors 23.6%","Open-set GAN rejects unseen gestures, improving myoelectric control by 23.6%","Unknown gesture? GAN says no, cutting myoelectric errors 23.6%","Myoelectric control gets GAN guard, 23.6% fewer errors","GAN guard for myoelectric control cuts errors 23.6%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000579,"raw_usage":{"total_tokens":2698,"prompt_tokens":887,"completion_tokens":1811,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":1705}},"tokens_in":503,"tokens_out":1811,"duration_ms":10412,"temperature":1.0,"reasoning_tokens":1705,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:03:25.814719+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the CNN and GAN on one subject's known gestures, then test the trained discriminator on real unknown gestures recorded from a different subject or on a different day; if the discriminator's AUC falls to roughly 0.5 or the active error rate stays above 80%, the synthetic-unknown proxy has failed. A sharper test is to withhold all real unknown gestures from threshold selection and set the threshold using only the GAN's synthetic samples; if the rejection performance then collapses to the no-discriminator baseline, the claimed gain depends on validation-time access to real unknowns rather than on the GAN's synthetic distribution.","supporting_citations":[{"cited_title":"Electromyography data for non-invasive naturally-controlled robotic hand prostheses,","cited_arxiv_id":null,"evidence_quote":"Supplies the Ninapro DB1 public dataset used for the main ratio and cross-domain experiments."},{"cited_title":"Generative adversarial nets,","cited_arxiv_id":null,"evidence_quote":"Introduces the GAN formulation whose generator and discriminator losses the paper adapts."},{"cited_title":"Open set learning with counterfactual images,","cited_arxiv_id":null,"evidence_quote":"Establishes the idea of using a GAN generator to produce synthetic outlier data for training an open-set discriminator."},{"cited_title":"A real-time pattern recognition based myoelectric control usability study implemented in a virtual environment,","cited_arxiv_id":null,"evidence_quote":"Motivates active error rate as the usability metric by arguing that halting a wrong action costs less than executing it."},{"cited_title":"Confidence-based rejection for improved pattern recognition myoelectric control,","cited_arxiv_id":null,"evidence_quote":"Defines confidence-based rejection after LDA and provides the performance-measure framework the paper extends to unknown gestures."},{"cited_title":"Selective classifi- cation for improved robustness of myoelectric control under nonideal conditions,","cited_arxiv_id":null,"evidence_quote":"Provides the earlier selective-classification baseline that rejects low-confidence outputs to reduce AER."},{"cited_title":"Deep learning with convo- lutional neural networks applied to electromyography data: A resource for the classification of movements for prosthetic hands,","cited_arxiv_id":null,"evidence_quote":"Documents the inter-subject/session variability in EMG recognition (about 66% accuracy across repetitions) that motivates the robustness problem."}],"review_version":1}