{"id":"36f357bc-43a9-4949-be0b-d503ed768d86","arxiv_id":"2412.17833","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Dense Poisson Disk Sampling before adaptive fine-tuning improved P300 BCI accuracy by about 5 percentage points and cut training time by 61%, but the evaluation protocol makes the gain unreliable.","lead":"The paper applies Poisson disk sampling to select EEG training examples before adaptive fine-tuning of a CNN for P300 brain-computer interface spelling, and reports higher accuracy and lower training time across two labs' data. Independent scrutiny is needed because the sampling budget and the highlighted fine-tuning rate were chosen using the same test data that produced the reported gains.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 5.36% gain at 40% adaptation does not match the paper's own tables, which show 4.49 percentage points; the 5.36 figure only matches a comparison across different adaptation rates.","rationale":"The reader's weakest assumption (test-set-based selection of the sampling factor) is a valid methodological concern and is part of my critique. However, the most load-bearing issue is more direct: the paper's headline quantitative claim, 'improved by 5.36% and standard deviation reduced by 12.22%', is not reproducible from the paper's own reported tables at the stated 40% adaptation rate. The tables give a 4.49 percentage point accuracy difference at 40% and a standard deviation reduction of 12.22 percentage points (not percent). The 5.36% figure matches only a mismatched comparison across adaptation rates. This internal inconsistency means the central numeric claim is unsupported by the evidence as presented, regardless of the selection-bias issue. The baseline instability further weakens the claim because the mean improvement is inflated by a few catastrophic w/o AS runs. These findings reinforce the reader's REJECT verdict; no verdict adjustment is needed.","tokens_in":17122,"tokens_out":9035,"duration_ms":63747,"concrete_test":"Recompute the means and standard deviations in Tables VI and VII at each adaptation rate from the per-subject entries, and compare the 40% row to the claimed 5.36% and 12.22%. Then rerun the subject-adaptive comparison at 40% with the sampling factor selected via an inner validation split (e.g., choose the sample size on a held-out validation subject, not on the test subjects). If the recomputed gain is ~4.5 rather than 5.36, or if the nested-selection gain drops below 1 percentage point or changes sign, the abstract's specific numeric claim and the implied robustness are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the abstract and Section V-D is that for OE+ME, subject-adaptive at a 40% adaptation rate, Active Sampling improves averaged classification accuracy by 5.36% and reduces standard deviation by 12.22%. The paper's own tables do not support this. Table VI (w/o AS) at 40% adaptation gives mean accuracy 78.13% and std 17.44%; Table VII (w/ AS) at 40% gives mean 82.62% (recomputed from per-subject entries: 82.65%) and std 5.22%. The accuracy difference is 4.49 percentage points, not 5.36. The value 5.36 arises only if one compares w/o AS at 40% (78.13) to w/ AS at 50% (83.49). The 'standard deviation reduction of 12.22%' is actually the absolute difference 17.44 - 5.22 = 12.22 percentage points, not a relative reduction (which would be about 70%). Even if this were a typographical error, the directional claim is fragile: the mean improvement is driven largely by subjects whose w/o AS baseline collapsed (e.g., S02 at 19.63%, S08 at 60.06%, S16 at 67.29% at 40%), while w/ AS pulls them back to near-normal levels. The reported benefit may therefore reflect stabilization of an unstable baseline rather than a genuine sampling advantage. Additionally, Section V-A states the 1200-sample factor was chosen by maximizing subject-independent test accuracy on the same subjects later used in the adaptive evaluation, so the reported gain is optimistically biased by test-set peeking. These issues together undermine the quantitative central claim as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an Active Sampling (AS) method based on Dense Poisson Disk Sampling (PDS) for adaptive transfer learning in P300-based brain-computer interfaces. The method is evaluated on two EEG datasets collected in different countries, one public and one newly replicated, with both healthy and patient populations. The authors report that AS improves averaged classification accuracy by 5.36% and reduces standard deviation by 12.22% at a 40% adaptation rate, while also reducing training time by about 61%. The paper includes subject-dependent, subject-independent, and subject-adaptive classification experiments, together with computational time and bitrate analyses.","tokens_in":17547,"tokens_out":7331,"duration_ms":61733,"significance":"If the reported results were valid, the proposed AS method would be a valuable contribution to reducing BCI calibration time in heterogeneous, multi-centre settings. The paper's strengths include the use of two distinct datasets with patient participants, a multi-centre replication design, and a clear focus on computational cost. However, the evaluation protocol contains a test-set selection bias, the headline numbers are inconsistent with the paper's own tables, and the without-AS baseline exhibits near-chance collapses that drive the reported improvement. These issues mean the central quantitative claim is not currently supported.","major_comments":[{"comment":"The sampling factor (1200 samples per subject) is selected by maximizing subject-independent test accuracy on the same subjects that are later used in the subject-adaptive comparison. Step 3 of the selection procedure states 'The best subject-independent classification accuracy test and its corresponding sample size were chosen.' This is test-set peeking: the hyperparameter is chosen using the test performance, so the reported improvements with AS are optimistically biased and cannot be interpreted as an unbiased estimate of the method's advantage.","section":"Section V-A, Table III"},{"comment":"The abstract claims a 5.36% accuracy improvement at a 40% adaptation rate. According to Table VI, the mean accuracy without AS at 40% is 78.13%, and according to Table VII, the mean accuracy with AS at 40% is 82.62%, a difference of 4.49 percentage points. The value 5.36 arises only from comparing w/o AS at 40% (78.13%) with w/ AS at 50% (83.49%). The reported 'standard deviation reduction of 12.22%' is also the absolute difference between 17.44% and 5.22% (in percentage points), not a relative reduction (which would be about 70%). The headline numbers need to be corrected to match the data.","section":"Abstract and Section V-D, Tables VI and VII"},{"comment":"The subject-adaptive baseline without AS collapses to near-chance levels for several subjects (e.g., S02 at 19.63% at 40%; S08 at 60.06% at 40% and 17.65% at 80%). The mean improvement with AS is therefore driven largely by stabilizing these collapsed runs rather than by a consistent sampling advantage. The paper should report per-subject results with repeated training runs and demonstrate that AS also improves accuracy for subjects whose without-AS baseline is not collapsed.","section":"Table VI"},{"comment":"The claim that Dense PDS is stable and consistent is not supported in the main text. The discussion refers to multiple repetitions in supplementary Table S1, but the number of repetitions, the variance across runs, and the specific results are not reported in the paper. Without these details, the reader cannot assess the statistical significance or robustness of the reported differences.","section":"Section VI"}],"minor_comments":[{"comment":"The term 'Poison Disk Sampling' should be 'Poisson Disk Sampling' (the authors consistently misspell 'Poisson').","section":"Throughout"},{"comment":"The text refers to the second dataset as 'IE' in the phrase 'using 1200 samples across both datasets (OE and IE)', but the dataset is named 'ME' elsewhere; please correct this typo.","section":"Section V-E"},{"comment":"The caption says 'Boxplot of training time' but the figure is a violin plot; the caption should match the figure type.","section":"Fig. 11 caption"},{"comment":"The dataset labels 'OE' and 'ME' appear as row entries in the table, which makes the grouping of subjects unclear; consider placing the dataset in a separate column.","section":"Table VIII"},{"comment":"Reference [24] is a general review of deep learning for EEG; the original Deep4Net architecture paper (Schirrmeister et al., 2017) should be cited instead or in addition.","section":"References"},{"comment":"The sentence 'the average classification accuracy for a subject adaptive with 40% of the adaptation rate in OE+ME w/o AS outperformed OE+ME w/AS with 5.36%' has the comparison direction reversed; it should read 'w/ AS outperformed w/o AS'.","section":"Section V-D"}],"recommendation":"reject","confidential_remarks":"The test-set peeking in the sampling-factor selection is a serious methodological flaw that invalidates the primary comparison. Combined with the inconsistency between the abstract and the tables, and the collapse of the without-AS baseline for several subjects, the central claim is not supported by the evidence as presented. These are not merely presentation issues; they require a re-analysis of the experimental protocol to be corrected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core quantitative claim in the abstract – 5.36% improvement at 40% adaptation – does not match the numbers in Tables VI and VII. The difference at 40% is 4.49 percentage points; the 5.36 figure only matches if you compare w/o AS at 40% to w/ AS at 50%. Similarly, the 'standard deviation reduced by 12.22%' is the absolute difference in stds, not a relative reduction. So the headline is wrong as stated.\n\nWhat's genuinely new: applying Dense PDS to adaptive transfer learning for P300 BCIs is a reasonable extension of the Zhang et al. line of work, and the ME dataset (a replication of OE with different hardware, sites, and patients) is a useful resource. The bitrate analysis is a good idea. The paper is honest enough to describe its sample-factor selection procedure, which is how we can see the problem.\n\nThe soft spots are serious. The sampling factor (1200) was chosen by maximizing subject-independent accuracy on the same subjects later used in the adaptive comparison. That's test-set peeking, so the reported gains are optimistic. The without-AS baseline has near-chance collapses at several adaptation rates (e.g., S02 at 40% with 19.63%, S08 at 80% with 17.65%), and AS pulls those back to normal. That means part of the 'improvement' is stabilization of an unstable baseline, not necessarily a sampling advantage. Also, the main adaptive results appear to be single runs; the supplementary repetition study is only for subject-independent classification, so we don't know the run-to-run variance of the headline numbers. On the plus side, the paper doesn't oversell in the discussion – it sticks to reporting what it did – but the abstract and Section V-D make a claim the tables don't support.\n\nI'd send this to a serious referee, but with a clear mandate to demand a corrected evaluation: hold out the test subjects entirely for hyperparameter selection, report repeated runs for the adaptive scheme, and fix the numerical errors. The underlying idea is plausible and the multi-centre dataset is valuable. As it stands, the evidence doesn't support the central claim, but the paper is a candidate for a solid revised version rather than a dead end.","headline":"The paper's headline improvement is a misread of its own tables, and the sampling factor was chosen on the test set; the multi-centre dataset and the core idea still deserve a proper evaluation.","tokens_in":18012,"tokens_out":2749,"would_cite":false,"duration_ms":21692,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Active sampling raises P300 BCI accuracy by 5.36% and cuts training time by 61%.","keywords":["P300 brain-computer interface","transfer learning","active sampling","Poisson disk sampling","adaptive fine-tuning","EEG classification","multi-centre data","deep learning"],"falsifier":"Re-run the subject-adaptive comparison with the sampling factor selected on a validation split that is never used for the final test, and check whether the 5.36% accuracy gain, the 12.22% standard-deviation reduction, and the 61% time saving persist; if they shrink or vanish, the reported advantage is an artifact of using the test set to choose the sample size.","tokens_in":16939,"feed_emoji":"🧠","tokens_out":5599,"duration_ms":44322,"temperature":0.7,"pith_summary":"The paper claims that a sampling strategy called Active Sampling, based on Dense Poisson Disk Sampling, makes transfer learning for P300 brain-computer interfaces faster and more accurate in heterogeneous, multi-centre conditions. Applied to a convolutional network (Deep4Net) with adaptive fine-tuning, the method improved average subject-adaptive classification accuracy by 5.36 percentage points and reduced inter-subject standard deviation by 12.22% compared with training on full data, while cutting CPU training time by roughly 61%. The authors argue this matters because real-world BCI use must cope with different equipment, countries, and health states, not uniform single-site data. If the claim holds, calibration of a P300 speller for a new user could be completed with far fewer trials and more stable performance.","feed_headline":"Active sampling lifts P300 BCI accuracy by 5.36%","feed_subtitle":"Poisson-disk sampling of EEG trials also cuts training time by 61% across multi-centre datasets.","key_machinery":"The load-bearing mechanism is Dense Poisson Disk Sampling (Dense PDS), a repulsive point process that generates diverse mini-batches (here, a whole sampled dataset of 1200 trials per subject) by requiring any two selected points to be at least a disk radius apart. Unlike random sampling, PDS makes the second-order product density vanish for nearby points, which drives the variance term in stochastic gradient descent negative and accelerates convergence. The 'Dense' variant draws points according to a categorical distribution over a mingling index, so the decision boundary between P300 and non-P300 classes is densely sampled. This sampled set is then used in Adaptive Transfer Learning (ATL): a Deep4Net CNN is pre-trained on source subjects' sampled data, its first convolutional layer is frozen, and the remaining layers are fine-tuned on the target subject's sampled data in 10% increments of the adaptation rate.","core_discovery":"The central discovery is that deliberately sampling a diverse, non-redundant subset of EEG trials—rather than using all available data—improves both accuracy and stability of adaptive transfer learning for P300 detection. The authors introduce Active Sampling (AS), a Dense Poisson Disk Sampling procedure that selects 1200 samples per subject, balancing P300 and non-P300 trials near the decision boundary through a mingling index. When a Deep4Net CNN pre-trained in a subject-independent manner is fine-tuned on this sampled target data, the averaged subject-adaptive accuracy rises by 5.36% and the standard deviation drops by 12.22% on the combined original and replicated multi-centre datasets. The method also reduces CPU training time by 61.17% and is presented as a general mechanism for reducing stochastic gradient noise and overfitting during domain adaptation.","pith_inferences":["Because the sampling factor (1200) was chosen by maximizing subject-independent test accuracy on the same subjects later used in the adaptive evaluation, the reported 5.36% gain and 61% time reduction are likely optimistic; an unbiased nested validation could lower them.","The same Dense PDS active-sampling scheme could be applied to other ERP-based BCIs (e.g., N170, error-related potentials) and to imbalanced classification beyond EEG, wherever stochastic gradient noise and redundant data slow convergence.","Combining Active Sampling with generative data augmentation, which the authors mention as future work, might push the accuracy gain further—but this is not tested in the paper.","The multi-centre replication design itself is a template: reproducing a published P300 protocol in a different country with different hardware is a strong stress test for transferability claims in BCI research."],"forward_implications":["A new BCI user could be calibrated with roughly 1200 sampled trials per subject instead of the full session data, cutting setup time while maintaining or improving accuracy.","Multi-centre EEG data from different countries, amplifiers, and patient groups can be pooled for transfer learning without the usual performance collapse, since sampling reduces inter-subject variance.","The 12.22% reduction in standard deviation means subject-adaptive P300 classification becomes more predictable across individuals, which matters for clinical assistive devices.","Training time drops by about 61%, so repeated recalibration during a session becomes feasible in real-world BCI use.","The approach yields higher bitrates at small numbers of stimulus blocks, meaning faster communication for spelling."],"supporting_citations":[{"why":"Supplies the Dense Poisson Disk Sampling procedure and the mingling index used for Active Sampling.","marker":"[20]"},{"why":"Supplies the adaptive transfer learning fine-tuning scheme of freezing the first convolutional layer and adjusting the rest.","marker":"[10]"},{"why":"Provides the original clinical P300 dataset (OE) used as source and target data.","marker":"[22]"},{"why":"Provides the replicated multi-centre dataset (ME) recorded with different equipment and subjects.","marker":"[23]"},{"why":"Supplies the Deep4Net CNN architecture used as the base model for all experiments.","marker":"[24]"},{"why":"Motivates variance reduction in SGD via repulsive point processes, the theoretical basis for the sampling approach.","marker":"[19]"}],"fun_headline_variants":["Active sampling boosts P300 BCI transfer learning accuracy 5.36%","Smarter EEG data sampling lifts BCI accuracy and cuts training time","P300 BCI: Poisson-disk sampling improves transfer learning by 5.36%","Adaptive sampling sharpens P300 BCI models across health states","Less data, better BCI: active sampling lifts accuracy 5.36%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper picks the 1200-sample factor by choosing the size that gives the best subject-independent test accuracy on the same subjects that later appear in the reported adaptive results, so the headline gains assume this selection does not leak test information into the method choice.","fun_headline_variants_meta":{"raw":{"variants":["Active sampling boosts P300 BCI transfer learning accuracy 5.36%","Smarter EEG data sampling lifts BCI accuracy and cuts training time","P300 BCI: Poisson-disk sampling improves transfer learning by 5.36%","Adaptive sampling sharpens P300 BCI models across health states","Less data, better BCI: active sampling lifts accuracy 5.36%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000961,"raw_usage":{"total_tokens":4108,"prompt_tokens":976,"completion_tokens":3132,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":3031}},"tokens_in":592,"tokens_out":3132,"duration_ms":17447,"temperature":1.0,"reasoning_tokens":3031,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:34:02.407440+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the subject-adaptive comparison with the sampling factor selected on a validation split that is never used for the final test, and check whether the 5.36% accuracy gain, the 12.22% standard-deviation reduction, and the 61% time saving persist; if they shrink or vanish, the reported advantage is an artifact of using the test set to choose the sample size.","supporting_citations":[{"cited_title":"Active mini-batch sampling using repulsive point processes,","cited_arxiv_id":null,"evidence_quote":"Supplies the Dense Poisson Disk Sampling procedure and the mingling index used for Active Sampling."},{"cited_title":"Adaptive transfer learning for eeg motor imagery classification with deep convolutional neural network,","cited_arxiv_id":null,"evidence_quote":"Supplies the adaptive transfer learning fine-tuning scheme of freezing the first convolutional layer and adjusting the rest."},{"cited_title":"An efficient p300-based brain–computer interface for disabled subjects,","cited_arxiv_id":null,"evidence_quote":"Provides the original clinical P300 dataset (OE) used as source and target data."},{"cited_title":"Perfor- mance evaluation of a p300 brain-computer interface using a kernel extreme learning machine classifier,","cited_arxiv_id":null,"evidence_quote":"Provides the replicated multi-centre dataset (ME) recorded with different equipment and subjects."},{"cited_title":"Deep learning techniques for eeg signal applications–a review,","cited_arxiv_id":null,"evidence_quote":"Supplies the Deep4Net CNN architecture used as the base model for all experiments."}],"review_version":1}