{"id":"4552e251-422f-4336-9f8e-4400b231011b","arxiv_id":"2507.11893","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A learnable modulate-demodulate resampling scheme reduces aliasing in downsampling and improves semantic segmentation by up to +3.3 mIoU.","lead":"The paper proposes Spatial Frequency Modulation, a module that resamples feature maps non-uniformly before downsampling and restores them during upsampling, to reduce aliasing and preserve fine details in semantic segmentation. It reports consistent mIoU gains across CNN and transformer segmentation models, plus gains in classification and adversarial robustness.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The frequency-modulation mechanism is not isolated: ARS alone hurts accuracy (Table 14: -0.5), MSAU alone provides most of the gain (+1.3), so the aliasing explanation lacks causal support.","rationale":"The paper's headline is not merely an empirical recipe; the mechanistic claim is what makes it novel. If the mechanism is wrong, the method could still be useful as an upsampling/refinement module, but the central claim and its theoretical motivation must be revised. The reader flagged the same causal-attribution issue, and my read agrees. I also note secondary inconsistencies (e.g., abstract says InternImage-T +1.4 while Table 3 reports +1.3/+1.5) and the unimplemented Eq. (3), but these are not the most load-bearing point. A single clean control can settle whether ARS's adaptive resampling is required for the gain. Because the empirical results are otherwise broad and the code is promised, the conditional verdict remains appropriate rather than outright rejection or acceptance.","tokens_in":33922,"tokens_out":7575,"duration_ms":86413,"concrete_test":"On the Cityscapes ResNet-50/PSPNet setup of Table 14, run a uniform-control experiment: keep MSAU and all losses and hyperparameters identical to SFM, but replace ARS with identity coordinate mapping (uniform grid, no adaptive offsets), so downsampling is the standard strided convolution. If this control achieves an mIoU within about 0.5 of SFM's 76.0, then the adaptive frequency-modulation component is not necessary for the gain; if its aliasing ratio also stays near the baseline 42.7% while mIoU improves, the aliasing explanation is refuted. To cover the opposite direction, retain ARS but replace MSAU with bilinear upsampling; Table 14's ARS + DySample (-0.2) already suggests the modulation alone cannot explain the result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SFM improves segmentation by frequency modulation: ARS shifts high-frequency content below the Nyquist rate before downsampling and MSAU demodulates it back. This causal story is the load-bearing element, and the paper's own experiments undercut it. In Table 14, ARS alone decreases mIoU by 0.5, MSAU alone increases it by 1.3, and the combination increases it by 3.3. Thus the component that performs modulation is harmful in isolation, while the upsampling-refinement component provides most of the benefit. The theoretical condition in Eq. (3) is not implemented; Section 3.2 states that the precise adaptive coordinates are difficult to compute, and instead the model is trained with L_FM (Eq. 12), which directly penalizes high-frequency power above Nyquist. That is operationally a low-pass constraint, not a lossless frequency shift, so the 'preserve rather than discard' claim is not established. Table 13 reports that SFM lowers the aliasing ratio from 42.7% to 34.2%, but this is measured on the full pipeline and cannot separate ARS's contribution from MSAU's smoothing. Without a control that keeps MSAU and removes or neutralizes ARS, the observed correlation between aliasing ratio and accuracy does not support the proposed mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Spatial Frequency Modulation (SFM), a lightweight module pair for semantic segmentation. Adaptive Resampling (ARS) is inserted before downsampling layers to non-uniformly resample high-frequency regions, allegedly shifting their spectral content below the Nyquist frequency; Multi-Scale Adaptive Upsampling (MSAU) replaces standard bilinear upsampling to demodulate the modulated features and refine predictions. The authors report consistent mIoU gains across multiple backbones and segmentation heads, including +1.5 mIoU on Mask2Former-Swin-T and +1.4 mIoU on UPerNet-InternImage-T on ADE20K, and +3.3 mIoU on Cityscapes with a ResNet-50 PSPNet baseline. They also extend the method to image classification, adversarial robustness, instance segmentation, and panoptic segmentation. The paper includes frequency-domain analysis, an ablation study, and comparisons with low-pass pooling, deformable convolution, and learned upsampling methods.","tokens_in":34183,"tokens_out":4330,"duration_ms":50369,"significance":"If the reported gains are reproducible, SFM would be a broadly applicable, inexpensive plug-in for dense prediction, and the paper's open-source release is a concrete asset. The experimental breadth is a genuine strength: the method is evaluated across FCN/PSPNet/CCNet/OCNet/PCAA heads, ResNet/Swin/ConvNeXt/InternImage backbones, and several tasks. However, the central causal claim that frequency modulation before downsampling is the driver of the gains is not established by the paper's own ablations. The component that performs the modulation (ARS) is harmful in isolation, the component that refines during upsampling (MSAU) provides most of the gain, and the frequency analysis largely reports the effect of an explicit loss term rather than an independent phenomenon. The paper would be substantially stronger if the mechanism were isolated with appropriate controls and if the relationship to the authors' prior ICLR 2024 work [50] were clarified.","major_comments":[{"comment":"The ablation does not support the causal attribution of the gains to the frequency-modulation mechanism. Table 14 shows ARS alone reduces mIoU by 0.5 (72.2 vs. 72.7), MSAU alone improves mIoU by 1.3 (74.0), and the combination improves by 3.3 (76.0). Since MSAU alone already delivers most of the improvement and ARS alone is harmful, the +3.3 gain cannot be attributed to the modulation/demodulation cycle without a control that keeps MSAU while neutralizing ARS (e.g., uniform resampling at the same cost, or an ARS with identity coordinates). The paper's own interpretation in §4.7 that 'ARS and MSAU cannot function independently' is exactly the claim that needs to be tested, not assumed. Please also report the aliasing ratio for each ablation row, so the reader can see whether mIoU tracks the aliasing ratio within the ablation rather than only in the full-pipeline comparison of Table 13.","section":"§4.7, Table 14"},{"comment":"The frequency analyses in Figures 10 and 11 are not independent evidence for the 'modulate, don't discard' claim, because the training objective L_FM in Eq. (12) directly penalizes the exact quantity being measured: spectral power above the Nyquist frequency. Figure 10(a) therefore confirms that the optimizer minimized a loss term, not that ARS implements the frequency-scaling operation described by Eq. (3). In fact, Section 3.2 states that the precise adaptive sampling coordinates for Eq. (3) are difficult to compute, and the implemented system is a learned coordinate map supervised by L_FM and L_SHF. To establish that high-frequency information is preserved rather than discarded, the paper should either demonstrate that the suppressed high-frequency components are recoverable after demodulation (e.g., by correlating the recovered spectrum with the original high-frequency content), or provide a control with L_FM removed. Table 18 shows that removing L_FM costs only 0.3 mIoU (74.7 vs. 74.4), which further suggests the aliasing-ratio reduction is not the main source of the improvement.","section":"§3.2, Eq. (12), and §3.4, Figures 10/11"},{"comment":"The motivating observation is correlational: the authors plot segmentation accuracy versus the aliasing ratio in existing models and observe an inverse relationship. This is presented as evidence that aliasing 'leads to' degradation, but no interventional experiment is performed on the baseline models to establish causation. The issue is compounded by the fact that SFM explicitly trains toward a lower aliasing ratio, so the correlation in Table 13 (42.7% to 34.2% AR with +3.3 mIoU) could reflect a common cause or a side effect of the upsampling refinement rather than a causal pathway. A concrete test would be to train SFM with the FM loss disabled (as in Table 18, row 2) and report the resulting aliasing ratio; if mIoU improves without a corresponding AR reduction, the central claim would be falsified.","section":"§3.1, Figure 1 and Figure 5"}],"minor_comments":[{"comment":"There is a typo in 'post-upssampled' near the end of Section 1; it should be 'post-upsampled'.","section":"§1, paragraph 2"},{"comment":"The text contains an inserted editorial marker 'Proofread:' immediately after 'Comparison with state-of-the-art on ADE20K.' This appears to be a leftover annotation and should be removed before submission.","section":"§4.3, after Table 3"},{"comment":"The header of Table 7 uses 'SMF' in the 'Ours' rows; this should be 'SFM' for consistency with the method name used throughout the paper.","section":"Table 7"},{"comment":"The heading 'Discusion' is misspelled; it should be 'Discussion'.","section":"§4.5 heading"},{"comment":"Reference [50] is the authors' own ICLR 2024 paper 'When Semantic Segmentation Meets Frequency Aliasing.' The submission should state explicitly what is new relative to that paper, particularly whether ARS or MSAU already appeared there, so that the novelty of the present TPAMI submission is unambiguous.","section":"§2, related works and reference [50]"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be a journal extension of the authors' ICLR 2024 paper [50], and the current text does not clearly delineate the delta. Given the identical first-author list and overlapping topic, the editor should require a clear novelty statement. The inserted 'Proofread:' marker in §4.3 also suggests the paper was not fully cleaned before review. The central mechanism question is substantive but addressable with additional controls, so I would not reject; major revision is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the empirical case is broad: consistent mIoU gains across ResNet, Swin, ConvNeXt, InternImage backbones on Cityscapes, Pascal Context, ADE20K, plus classification, adversarial robustness, and instance/panoptic segmentation. The code is available, and the modules are simple to drop into existing architectures. Second, the paper's own ablation undermines the central mechanism. Table 14 shows ARS alone drops mIoU by 0.5, MSAU alone adds 1.3, and the combination adds 3.3. The component that is supposed to reduce aliasing is harmful in isolation, so the 'aliasing degradation' story is not causally supported.\n\nWhat is actually new: combining adaptive resampling with non-uniform upsampling and cascaded multi-scale refinement for full-resolution segmentation. That combination appears to work. The RDF plots and feature visualizations are consistent with the training objective, and the computational overhead is minor.\n\nThe soft spots, in proportion. The frequency analysis is not an independent effect: the FM loss explicitly penalizes high-frequency power above Nyquist, so Figures 10 and 11 just confirm the objective. Equation (3) is acknowledged as not implemented. The paper says SFM preserves rather than discards high frequencies, but operationally it is a learned low-pass with a learned upsampling decoder. That is fine, but the narrative oversells it.\n\nThe bigger problem is novelty. Reference [50] is the authors' own ICLR 2024 paper, 'When semantic segmentation meets frequency aliasing', which appears to study exactly the relationship between aliasing and segmentation accuracy. The current paper does not cite it and claims to investigate this 'for the first time.' That is not defensible and needs to be resolved.\n\nAlso, numbers are inconsistent across tables: Mask2Former-Swin-T is +1.5 in the abstract, +1.4 in Table 3, +1.5 in Table 10. Minor, but sloppy.\n\nWho this is for: practitioners who want a drop-in module for dense prediction. The engineering result is plausible and worth a serious referee, but the referee should ask for a control that isolates ARS from MSAU, a softened causal claim, and an honest positioning against the prior work. I would send it out, but I would not let the current discussion of the mechanism stand without revision.","headline":"Broad, plausible engineering gains and a lightweight module, but the paper's causal aliasing story is undone by its own ablation and the novelty claim overlooks the authors' ICLR 2024 paper.","tokens_in":34757,"tokens_out":5768,"would_cite":false,"duration_ms":59238,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Spatial Frequency Modulation improves semantic segmentation by shifting high-frequency features below the Nyquist rate before downsampling and recovering them during upsampling.","keywords":["spatial frequency modulation","aliasing","semantic segmentation","adaptive resampling","non-uniform upsampling","frequency learning","anti-aliasing","dense prediction"],"falsifier":"A direct test would be to compare segmentation accuracy on features that have been deliberately low-pass filtered (removing high frequencies) versus features modulated to lower frequencies without discarding them; if the low-pass version does not degrade as much as predicted, the causal role of aliasing in the SFM gains is weaker.","tokens_in":1195,"feed_emoji":"🎯","tokens_out":4687,"duration_ms":95317,"temperature":0.7,"pith_summary":"The paper argues that a large part of the accuracy lost in modern segmentation networks is caused by aliasing: when feature maps are downsampled, energy above the Nyquist frequency folds into lower frequencies and distorts the features. To avoid this without throwing away fine details, the authors propose Spatial Frequency Modulation (SFM), which non-uniformly resamples the feature map before downsampling so high-frequency regions are stretched and shifted to lower frequencies, and then reverses the operation during upsampling. If correct, SFM is a lightweight add-on that improves a wide range of CNN and transformer segmentation models, including Mask2Former and InternImage, and also helps in classification, adversarial robustness, and instance/panoptic segmentation.","feed_headline":"Resampling cuts aliasing, boosting segmentation by up to 3.3 mIoU","feed_subtitle":"A lightweight modulate-demodulate cycle protects fine details during downsampling in CNNs and transformers.","key_machinery":"The load-bearing machinery is Adaptive Resampling (ARS) for modulation and Multi-Scale Adaptive Upsampling (MSAU) for demodulation. ARS generates an attention map via difference-aware convolution and pyramid spatial pooling, maps uniform coordinates to non-uniform ones so that high-attention (high-frequency) areas are sampled more densely, and thereby lowers the frequency of those signals in accordance with the Frequency Scaling Property; two extra losses, the frequency modulation loss and the semantic high-frequency loss, supervise this resampling. MSAU reverses the coordinate deformation with Delaunay triangulation and barycentric interpolation, then refines the prediction with cascaded Local Pixel Relation Modules whose dilation grows to capture multi-scale relations between densely and sparsely sampled regions.","core_discovery":"The central claim is that the \"aliasing degradation\" phenomenon—lower segmentation accuracy as the aliasing ratio of feature maps increases—can be countered by a modulate-demodulate cycle. The paper introduces the aliasing ratio as the proportion of spectral power above the Nyquist frequency, shows empirically that it correlates inversely with accuracy across ResNet, Swin, and ConvNeXt backbones, and then demonstrates that replacing uniform downsampling with adaptive resampling (ARS) and uniform upsampling with multi-scale adaptive upsampling (MSAU) consistently raises mIoU, with gains such as +1.5 mIoU for Mask2Former-Swin-T and +1.4 for InternImage-T on ADE20K.","pith_inferences":["If the causal story is right, the same modulate-demodulate cycle should help any dense-prediction task with repeated downsampling, such as depth estimation or optical flow, not just segmentation.","The paper's own ablation leaves open that a large share of the gain comes from the upsampling refinement: MSAU alone improves +1.3 mIoU while ARS alone degrades -0.5, so the causal role of the frequency modulation itself is not separately established.","A testable extension is to apply ARS to only the first downsampling layer, where aliasing is strongest, and measure whether the benefit scales with the number of protected stages.","The aliasing-ratio metric could be repurposed as a diagnostic tool for selecting which layers of a pretrained network most need anti-aliasing treatment."],"forward_implications":["Semantic segmentation models can gain accuracy without architectural redesign by adding SFM at downsampling and upsampling stages.","The gains concentrate on boundaries: boundary F-score, boundary IoU, and boundary accuracy all improve, and boundary-related errors such as false responses, merging mistakes, and displacements drop.","SFM combines with existing anti-aliasing and refinement methods such as FLC, CondConv, and SegFix, producing further improvements.","Because the modulation is a resampling operation rather than a filter, high-frequency details are preserved instead of discarded, which distinguishes SFM from low-pass pooling approaches.","The same modulation step transfers to image classification, adversarial robustness, and instance/panoptic segmentation, suggesting the mechanism is a general property of downsampling in vision networks."],"supporting_citations":[{"why":"Supplies the Shannon sampling theorem that frames high-frequency energy above the Nyquist rate as the source of aliasing.","marker":"[17]"},{"why":"Supplies the Nyquist criterion that defines the sampling-rate condition the modulation step tries to satisfy.","marker":"[18]"},{"why":"Gives the Frequency Scaling Property that dense sampling lowers a signal's frequency, the theoretical basis for ARS.","marker":"[26]"},{"why":"Prior work linking aliasing to adversarial vulnerability and a low-pass pooling baseline SFM is compared against.","marker":"[19]"},{"why":"Prior anti-aliasing method using learned blurring filters, used as a baseline and related empirical evidence.","marker":"[20]"},{"why":"Adaptive downsampling method for segmentation that SFM extends and compares with in low-resolution settings.","marker":"[29]"},{"why":"Supplies the saliency-weighted coordinate mapping formula that ARS adapts for non-uniform sampling.","marker":"[79]"},{"why":"PSPNet is the principal segmentation head used in the ablation studies that isolate ARS and MSAU contributions.","marker":"[31]"},{"why":"Mask2Former is the main transformer-based architecture improved by SFM in the ADE20K experiments.","marker":"[33]"}],"fun_headline_variants":["Modulation keeps fine details, sharpening segmentation by 3.3 mIoU","Frequency shift before downsampling keeps fine detail, lifts mIoU","SFM: adaptive resampling halts aliasing, boosts segmentation","Modulate-demodulate avoids aliasing, better segmentation","Resampling before and after downsampling sharpens segmentation"],"cache_read_input_tokens":36864,"weakest_assumption_plain":"The load-bearing premise is that aliasing—the folding of high-frequency energy above the Nyquist rate during downsampling—is a primary cause of the observed accuracy loss, and that the correlation between aliasing ratio and accuracy is causal.","fun_headline_variants_meta":{"raw":{"variants":["Modulation keeps fine details, sharpening segmentation by 3.3 mIoU","Frequency shift before downsampling keeps fine detail, lifts mIoU","SFM: adaptive resampling halts aliasing, boosts segmentation","Modulate-demodulate avoids aliasing, better segmentation","Resampling before and after downsampling sharpens segmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001099,"raw_usage":{"total_tokens":4582,"prompt_tokens":935,"completion_tokens":3647,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":3555}},"tokens_in":551,"tokens_out":3647,"duration_ms":32180,"temperature":1.0,"reasoning_tokens":3555,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:59:23.080942+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to compare segmentation accuracy on features that have been deliberately low-pass filtered (removing high frequencies) versus features modulated to lower frequencies without discarding them; if the low-pass version does not degrade as much as predicted, the causal role of aliasing in the SFM gains is weaker.","supporting_citations":[{"cited_title":"Learning to zoom: a saliency-based sampling layer for neural net- works,","cited_arxiv_id":null,"evidence_quote":"Supplies the saliency-weighted coordinate mapping formula that ARS adapts for non-uniform sampling."}],"review_version":1}