{"id":"b9fddc77-ce8d-403b-b6ea-f7c354b82491","arxiv_id":"2501.11538","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"DenoMAE adds noise as a fifth modality to a masked autoencoder and claims data-efficient denoising and modulation classification, but its efficiency claim is not yet supported by matched experiments.","lead":"DenoMAE is a transformer that pretrains on noisy radio signals and their constellation diagrams, treating noise as an extra input, then classifies modulation types with far fewer labeled samples. It claims state-of-the-art accuracy with one-tenth of the unlabeled data and one-third of the labeled data of prior methods, but the claim rests on uncontrolled comparisons.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA and data-efficiency claims rest on Table II, which compares DenoMAE across different datasets, class counts, and SNRs; no matched or from-scratch baseline is provided, so the central claim is unsupported.","rationale":"The reader's verdict is REJECT, and my independent reading identifies the same load-bearing concern: Table II is not a valid comparison. The authors compare DenoMAE on a self-generated dataset with 10 classes and SNR 0.5 dB to methods evaluated on different datasets, class counts, and SNR regimes. For a claim of state-of-the-art performance and data efficiency, a matched baseline on the same data is essential. The absence of any same-architecture from-scratch baseline is particularly problematic because it leaves open the possibility that the 83.5% accuracy reflects the dataset's ease or the architecture's inductive bias rather than the pretraining procedure. The claimed 22.1% improvement over a non-pre-trained baseline is mentioned but never shown, making the pretraining benefit unverifiable. I also note the internal inconsistency between the abstract ('10% and 3%') and the full text ('10x and 3x'), which further weakens the precision of the data-reduction claim. My concrete test would settle the concern by providing the missing baselines: a from-scratch ViT and a standard CNN trained on the same fine-tuning set, plus an evaluation on a public benchmark. If the from-scratch baseline performs comparably, the central claim fails; if DenoMAE clearly beats both, the claim gains support. Until such a test is run, the REJECT verdict stands.","tokens_in":1146,"tokens_out":983,"duration_ms":92288,"concrete_test":"Run DenoMAE and a from-scratch ViT-B encoder of identical architecture (random init, no pretraining) on the same 10-class, 1,000-sample fine-tuning dataset at SNR 0.5 dB, using the same classification head and training schedule. Also run a standard CNN baseline (e.g., the AlexNet-based constellation classifier of Ref. [34]) on the same data. Report mean plus/minus std over at least 3 seeds. If the from-scratch accuracy is within a few points of 83.5%, the pretraining and data-efficiency claims are unsupported; if DenoMAE's accuracy falls to or below these baselines on a public benchmark like RadioML 2018.01, the SOTA claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that DenoMAE achieves state-of-the-art 83.5% accuracy with only 10,000 pretraining and 1,000 fine-tuning samples, representing a dramatic data reduction. This claim rests on Table II, which compares DenoMAE on a self-generated 10-class constellation dataset at SNR 0.5 dB against AlexNet (2 classes, 800k samples), NMformer (10 classes, 106,800 samples), CNN-AMC (4 classes, -6 dB), and DL-GRF (4 classes, 0 dB). Accuracy in AMC depends strongly on class count, SNR, signal length, and dataset difficulty, so these numbers are not transferable. No same-architecture from-scratch baseline is reported on the authors' dataset or on a public benchmark; the claimed 22.1% improvement over a non-pre-trained baseline (Section V.B) is mentioned but never shown. Without a matched baseline, neither the 'state-of-the-art' nor the '10x/3x data reduction' claim is supported. The abstract even says '10% and 3%' rather than '10x and 3x', compounding the inconsistency. The central claim is therefore unverifiable from the evidence presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DenoMAE, a multimodal masked-autoencoder pretraining approach for automatic modulation classification. The model operates on five input modalities (noisy constellation image, noisy signal, clean constellation image, clean signal, and noise), masks 75% of patches, and reconstructs the masked content; the pretrained encoder is then fine-tuned with a two-layer MLP head for 10-class classification. The authors report 83.5% accuracy at 0.5 dB SNR and 77.5% at -10 dB using 10,000 pretraining and 1,000 fine-tuning samples, and claim a 10x reduction in unlabeled pretraining data and a 3x reduction in labeled fine-tuning data relative to prior methods, as well as a 22.1% accuracy gain over a non-pretrained model at low SNR.","tokens_in":8140,"tokens_out":7955,"duration_ms":88760,"significance":"If the empirical claims were substantiated, the paper would address a relevant problem: data-efficient automatic modulation classification at low SNR, with the interesting design choice of including noise as an explicit modality. The per-modality ablation in Table I provides useful evidence that the additional modalities contribute to downstream accuracy. However, the significance of the paper rests almost entirely on the validity of Table II and on the claimed 22.1% pretraining gain, and neither is supported by the evidence as presented. The manuscript does not provide a controlled comparison, a public benchmark evaluation, error bars, or a definition of the core noise modality, so the central contribution cannot currently be assessed.","major_comments":[{"comment":"The state-of-the-art claim is not supported by Table II: DenoMAE is evaluated on a private 10-class dataset at 0.5 dB SNR, while AlexNet uses 2 classes at 8 dB with 800,000 training samples, NMformer uses 10 classes on a different dataset at 0.5-4.5 dB, CNN-AMC uses 4 classes at -6 dB, and DL-GRF uses 4 classes at 0 dB. Accuracy in AMC depends strongly on class count, SNR, signal length, and dataset difficulty, so the 83.5% figure cannot be directly compared with these numbers. The derived claims of 80x fewer samples than AlexNet and 10.7x fewer than NMformer are therefore not meaningful without a controlled comparison on the same data, same class count, same SNR protocol, and ideally the same architecture family.","section":"Section V.E, Table II"},{"comment":"The claimed 22.1% improvement over the non-pretrained baseline at -10 dB SNR is unverifiable: the manuscript reports no from-scratch baseline, no architecture-matched supervised-only training result, and no table or figure for this comparison. Since the pretraining benefit is a central contribution, the authors need to include a same-architecture baseline trained from scratch on the same 1,000 labeled examples, with multiple random seeds, and preferably report this comparison at every SNR point shown in Figure 4.","section":"Section V.B"},{"comment":"The term unlabeled oversells the pretraining requirement. Although class labels are not used, the reconstruction targets are clean paired noiseless signals and constellation diagrams for each noisy input, so the pretraining data require paired clean versions of every sample. This is a form of signal-level supervision rather than purely unlabeled data. The data-efficiency claim should be restated to acknowledge this requirement, and the paper should explain how clean targets are obtained in a practical deployment setting.","section":"Section III; Section V.A"},{"comment":"The noise-as-explicit-modality, which is the main novelty of the paper, is never defined. The manuscript does not specify how the noise modality is constructed, for example whether it is the residual between the noisy and clean signals, a separate noise image, or a noise-power map, and no equation or algorithm for computing it is provided. This omission prevents reproduction of the method and leaves the core contribution untestable; the authors should specify exactly how each of the five modalities is computed.","section":"Section III-A; Section IV-A"}],"minor_comments":[{"comment":"The abstract says a 10% reduction in unlabeled pretraining data and a 3% reduction in labeled fine-tuning data, while the introduction and Section V.E claim one-tenth and one-third (10x and 3x) reductions; these statements are contradictory and should be corrected.","section":"Abstract; Section I; Section V.E"},{"comment":"The text mentions a 7x7 complex plane but the input images are 224x224; please clarify how the constellation samples are mapped to this grid and how the three exponential decay rates are chosen, since these are free parameters.","section":"Section IV.A"},{"comment":"The statement that our codes are public at GitHubDenoMAE does not include a URL or repository identifier; a complete link should be provided.","section":"Abstract; Section V.A"},{"comment":"All accuracy numbers are point estimates; please report means and standard deviations over multiple runs and include error bars in the figures.","section":"Table I; Figure 4"}],"recommendation":"reject","confidential_remarks":"To the editor: the central accuracy and data-efficiency claims are unsupported by the experimental design as submitted. The fix is not a local revision but a substantially new comparison study, including matched baselines and evaluation on a public benchmark or a released dataset. I would be open to reconsidering a future version that provides such evidence, since the underlying masked-modeling idea is plausible and the per-modality ablation is informative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"DenoMAE is a pretty direct adaptation of MultiMAE/4M to modulation signals and constellation diagrams, with one genuinely new twist: treating noise itself as an input modality. That is a sensible idea and the paper deserves credit for framing it clearly, citing the right prior work, and running a small ablation (Table I) that shows the modalities help in a graceful, monotonic way. The visual extrapolation to out-of-distribution low SNRs is also interesting.\n\nThe problem is the evaluation. The central claim—83.5% accuracy with 10x less unlabeled and 3x less labeled data—rests on Table II, which compares DenoMAE on 10 classes at 0.5 dB against AlexNet on 2 classes at 8 dB, NMformer on 10 classes at 0.5-4.5 dB, CNN-AMC on 4 classes at -6 dB, and DL-GRF on 4 classes at 0 dB. Class count, SNR, signal length, and dataset difficulty all move accuracy in AMC, so these numbers are not comparable. There is no same-architecture from-scratch baseline on the authors' own dataset, no error bars, and no public benchmark such as RadioML. The claimed 22.1% improvement over a non-pretrained baseline is stated in the text but never shown in a table or figure. The abstract even says '10%' and '3%' where the full text says '10x' and '3x', which is the kind of slip that makes me worry about how carefully the numbers were checked.\n\nThere are also smaller issues: 'unlabeled' pretraining still uses clean targets as reconstruction labels, so the data-efficiency claim is softer than it sounds; the code link is promised but no URL is given in the text; and the private dataset makes independent verification impossible.\n\nWho is this for? Researchers working on self-supervised pretraining for radio signals. They might get a useful idea from the noise-as-modality formulation, but they cannot trust the empirical claims as reported. This is a paper for peer review, not a desk reject, because the method is plausible and the community needs rigorous tests of multimodal masked modeling in this domain. But it needs major revision: matched baselines, a public dataset, error bars, and a corrected abstract.","headline":"Plausible application of MultiMAE to modulation signals, but the headline data-efficiency claims are unsupported by an uncontrolled comparison table.","tokens_in":8739,"tokens_out":3394,"would_cite":false,"duration_ms":33757,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DenoMAE reaches 83.5% modulation-classification accuracy from 10,000 unlabeled and 1,000 labeled samples by treating noise as an explicit input modality.","keywords":["multi-modality","vision transformer","modulation classification","constellation diagrams","denoising","masked autoencoder","self-supervised pretraining","low SNR"],"falsifier":"Train DenoMAE and a same-architecture classifier from scratch on the identical 10-class dataset at the same 0.5 dB SNR, giving both only 1,000 labeled samples; if the from-scratch classifier matches or exceeds 83.5% accuracy, the pretraining advantage is refuted. A second check is to retrain one of the comparison systems on DenoMAE's dataset and sample counts and see whether 10,000 pretraining and 1,000 fine-tuning examples still win.","tokens_in":7679,"feed_emoji":"📡","tokens_out":10480,"duration_ms":95631,"temperature":0.7,"pith_summary":"This paper asks whether a self-supervised autoencoder can clean noisy radio signals well enough that modulation classification requires almost no labeled data. The proposed answer, DenoMAE, pre-trains a masked autoencoder on five views of the same signal—noisy and noiseless waveforms, noisy and noiseless constellation diagrams, and an explicit noise view—so the model must reconstruct the clean views from the noisy ones. The paper reports that this pretraining lets a ten-way modulation classifier reach 83.5% accuracy using only 10,000 unlabeled and 1,000 labeled samples, with the largest gains at low signal-to-noise ratios and with denoising that still works below the SNR range seen in training. A sympathetic reader would take the paper's central bet to be that noise, treated as an input rather than only as corruption, is a reusable source of self-supervision for communication signals. If that bet pays off, radio-frequency classification becomes practical in settings where labeled captures are scarce and noise conditions are challenging.","feed_headline":"DenoMAE hits 83.5% accuracy with 1,000 labels","feed_subtitle":"A masked autoencoder pre-trains on noisy and clean signal views, then fine-tunes on a tiny labeled set for low-SNR classification.","key_machinery":"The load-bearing mechanism is multimodal masked reconstruction with noise as a co-equal modality. Each of the five input modalities is a three-channel 224-by-224 image, divided into 16-by-16 patches and linearly embedded; a twelve-layer transformer encoder processes only the 25% of patches left visible after random masking, a linear projection followed by layer normalization maps each modality into a shared latent space, and a four-layer decoder reconstructs all masked patches for all modalities from the concatenated shared code. The training loss is a weighted sum of mean-squared-error reconstruction losses across the five modalities. Because the decoder must regenerate the noiseless views from representations that also contain the noisy views, the shared latent space is where the model learns to attribute structure to signal and noise separately; downstream classification discards the decoder and uses only the encoder.","core_discovery":"On the paper's own terms, the central claim is that a masked autoencoder can be turned into a data-efficient modulation classifier by making noise a first-class input modality. DenoMAE is pre-trained on 10,000 unlabeled samples, each rendered as five paired views: a noisy constellation diagram (a scatter plot of a signal's amplitude and phase states), a noisy signal image, the corresponding noiseless constellation diagram, the noiseless signal image, and an explicit noise image. With 75% of patches randomly masked, a shared encoder must reconstruct all masked patches of all five views from the unmasked ones, which forces it to separate signal content from noise. After fine-tuning the encoder with only 1,000 labeled samples, the model reports 83.5% test accuracy across ten modulation classes at 0.5 dB SNR, 77.5% at -10 dB, a 22.1% gain over the same architecture without pretraining at that SNR, and meaningful denoising on SNRs down to -20 dB, below the pretraining range.","pith_inferences":["An implication the authors do not spell out is that the same noise-as-modality mechanism should transfer to other denoising tasks with paired clean and noisy observations, such as speech enhancement or radar clutter suppression, because nothing in the architecture is modulation-specific.","A fair test of the data-efficiency claim would be to hold one dataset fixed and vary only the number of pretraining and fine-tuning samples; the absolute gap might shrink, but the direction of the claim would then be established.","A testable extension suggested by the architecture is to vary the masking ratio per modality: if the noise view is truly load-bearing, downstream accuracy should be more sensitive to masking the noise view heavily than to masking any other single view heavily."],"forward_implications":["A modulation classifier for a previously unseen band could be built from roughly 10,000 unlabeled captures and 1,000 labeled examples, rather than the hundreds of thousands of samples used by earlier systems.","Pretraining matters most in the regime where supervised classifiers typically fail: at low SNR, where the reported gain over no pretraining is 22.1 percentage points at -10 dB.","Because the model denoises inputs at SNRs below its training range, a model pretrained on a convenient SNR window could be deployed in noisier environments without retraining.","The ablation result—accuracy rising from 81.3% with a single modality to 83.5% with all five—indicates that each added view, including the explicit noise view, contributes a small but consistent gain."],"supporting_citations":[{"why":"Supplies the masked-autoencoder reconstruction objective that DenoMAE extends to multiple modalities.","marker":"[27]"},{"why":"Provides the multimodal masked-pretraining design that DenoMAE adapts to signal and noise modalities.","marker":"[11]"},{"why":"Defines the enhanced constellation-diagram representation used as input and supplies one of the comparison baselines in Table II.","marker":"[34]"},{"why":"Supplies the CNN-based comparison baseline in Table II.","marker":"[35]"},{"why":"Supplies the image-classification comparison baseline in Table II.","marker":"[36]"}],"fun_headline_variants":["DenoMAE: Noise as input, 1k labels for 83.5% accuracy","DenoMAE pretrains on noise to classify with only 1,000 labels","Low-SNR modulation classification made data-efficient by DenoMAE","DenoMAE: multimodal denoising autoencoder for few-shot modulation recognition"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline numbers compare DenoMAE, tested on its own generated 10-class dataset at a signal-to-noise ratio of 0.5 dB, with earlier systems evaluated on different class counts, SNRs, and sample sizes; if those cross-paper numbers cannot be fairly compared, the claimed accuracy and data-efficiency advantage has no common baseline.","fun_headline_variants_meta":{"raw":{"variants":["DenoMAE: Noise as input, 1k labels for 83.5% accuracy","DenoMAE pretrains on noise to classify with only 1,000 labels","Low-SNR modulation classification made data-efficient by DenoMAE","DenoMAE: multimodal denoising autoencoder for few-shot modulation recognition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000934,"raw_usage":{"total_tokens":4000,"prompt_tokens":952,"completion_tokens":3048,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":2959}},"tokens_in":568,"tokens_out":3048,"duration_ms":21277,"temperature":1.0,"reasoning_tokens":2959,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:08:14.580828+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train DenoMAE and a same-architecture classifier from scratch on the identical 10-class dataset at the same 0.5 dB SNR, giving both only 1,000 labeled samples; if the from-scratch classifier matches or exceeds 83.5% accuracy, the pretraining advantage is refuted. A second check is to retrain one of the comparison systems on DenoMAE's dataset and sample counts and see whether 10,000 pretraining and 1,000 fine-tuning examples still win.","supporting_citations":[{"cited_title":"Automatic modulation classification using techniques from image classification,","cited_arxiv_id":null,"evidence_quote":"Supplies the image-classification comparison baseline in Table II."},{"cited_title":"Multimae: Multi- modal multi-task masked autoencoders,","cited_arxiv_id":null,"evidence_quote":"Provides the multimodal masked-pretraining design that DenoMAE adapts to signal and noise modalities."},{"cited_title":"Modulation classification based on signal constellation diagrams and deep learning,","cited_arxiv_id":null,"evidence_quote":"Defines the enhanced constellation-diagram representation used as input and supplies one of the comparison baselines in Table II."},{"cited_title":"Automatic modulation classification: A deep learning enabled approach,","cited_arxiv_id":null,"evidence_quote":"Supplies the CNN-based comparison baseline in Table II."}],"review_version":1}