{"id":"b7513b20-1d50-4c4a-9e8c-ce5ead8f9c30","arxiv_id":"2412.11076","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"MoRe regularizes class-patch attention with a directed graph module and a CAM-informed contrastive loss, improving weakly supervised semantic segmentation.","lead":"This paper presents MoRe, a training method that adds two regularizers to the class-patch attention of a Vision Transformer for weakly supervised semantic segmentation. On PASCAL VOC and MS COCO, MoRe improves pseudo-label quality and beats prior single-stage and some multi-stage methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MoRe's SOTA claim is confounded by ImageNet-21k pretraining: MoRe uses ViT-B/16 pretrained on ImageNet-21k while key baselines ToCo/DuPL use ImageNet-1k, so the 3.1% mIoU gain may come from initialization rather than the proposed regularizers.","rationale":"We chose the pretraining confound as the most load-bearing concern because the paper's central claim is explicitly comparative ('state-of-the-art', 'surpassing recent single-stage and even multi-stage methods'). The implementation explicitly cites Ridnik et al. (2021), which is ImageNet-21k pretraining, while typical baselines in this line of work use ImageNet-1k. Since WSSS is highly sensitive to the pretrained backbone, this is a concrete, identifiable threat to the comparative claim. The reader's weakest_assumption focused on LIR's reliance on CAM; that is a valid concern about mechanism attribution, but it does not undermine the performance comparison as directly. We partially agree with the reader: their rationale already mentions the pretraining issue, but their stated weakest assumption does not. We recommend keeping the conditional verdict, because the confound is testable and does not invalidate the internal ablations.","tokens_in":14592,"tokens_out":4957,"duration_ms":38687,"concrete_test":"Retrain ToCo and DuPL (or the strongest baseline) using the same ImageNet-21k pretrained ViT-B/16 backbone and, as far as possible, the same multi-class token and training recipe as MoRe; alternatively retrain MoRe with ImageNet-1k pretraining. Compare VOC val mIoU under matched initialization. If MoRe's margin over the best baseline shrinks to less than ~1 mIoU, the state-of-the-art claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"MoRe's implementation section states: 'MoRe adopts ViT-B/16 pretrained on ImageNet (Ridnik et al. 2021) as encoder.' Ridnik et al. is ImageNet-21k pretraining. The leading single-stage baselines in Table 2, ToCo and DuPL, use ViT-B/16 pretrained on ImageNet-1k (DeiT/standard ViT). ImageNet-21k initialization is known to improve WSSS mIoU by several points because it provides richer visual features and better attention priors. Therefore the reported 76.4 vs 73.3 mIoU over DuPL, and the claim of surpassing multi-stage methods, may largely reflect a pretraining advantage rather than the proposed GCR and LIR regularizers. The paper's ablations (Table 3) compare against its own baseline with the same 21k backbone, so they show internal efficacy, but they do not equate to SOTA against matched-pretraining baselines. This confound directly threatens the paper's headline claim. A secondary concern is that LIR uses CAM-derived masks as supervision, which can transfer CAM's biases into LAM; however, the pretraining mismatch is the more decisive issue for the SOTA comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"MoRe is a single-stage WSSS method built on a ViT with multi-class tokens. It introduces two regularizers for class-patch attention: a Graph Category Representation (GCR) module that models class-patch attention as a directed graph with dynamic neighbor selection and graph aggregation, and a Localization-informed Regularization (LIR) module that uses CAM-derived confident and uncertain relation masks to supervise contrastive losses (Lcre and Lure) on class-patch similarity. The paper reports 76.4 mIoU on PASCAL VOC val, 75.0 on VOC test, and 47.4 on COCO val, with pseudo masks reaching 80.0 mIoU, and claims state-of-the-art performance over both single-stage and multi-stage methods. Ablations show that each component contributes to the final result and that the method reduces confusion ratio relative to ToCo and DuPL.","tokens_in":14880,"tokens_out":2744,"duration_ms":24798,"significance":"If the claims hold, MoRe would be a useful contribution to LAM-based WSSS: it identifies a specific failure mode (artifact patches activated by class tokens), proposes two complementary and comparatively cheap regularizers, and ships an efficient single-stage pipeline with code. The paper also includes a class-specific confusion-ratio analysis that goes beyond plain mIoU, which is informative for diagnosing false-positive artifacts. However, the headline state-of-the-art claim is currently confounded by a pretraining mismatch with the main baselines, and the LIR module is partly supervised by CAM, so the evidence that MoRe resolves the artifact issue rather than inheriting CAM's biases is not yet conclusive. The ablations are internally consistent but all numbers come from single runs with validation-set hyperparameter tuning.","major_comments":[{"comment":"The state-of-the-art comparison is confounded by pretraining: MoRe adopts ViT-B/16 pretrained on ImageNet-21k (Ridnik et al. 2021), whereas the key single-stage baselines ToCo and DuPL in Table 2 use ViT-B/16 pretrained on ImageNet-1k, so the reported +3.1 mIoU over DuPL and the claim of surpassing multi-stage methods may derive from the stronger initialization rather than from GCR and LIR. Please add matched-pretraining baselines (e.g., ToCo and DuPL with ImageNet-21k ViT-B/16) or explicitly quantify the pretraining contribution by also running MoRe with an ImageNet-1k backbone.","section":"Experimental Settings (Implementation Details) and Table 2"},{"comment":"Because LIR supervises class-patch attention with CAM-derived masks from Eq. (6), the improvement of LAM over CAM is partly inherited from the teacher signal, so the claimed resolution of the artifact issue is not fully demonstrated. Please report an experiment where LIR is trained with pseudo-masks from an independent source (or where the CAM-derived masks are corrupted/removed), and compare MoRe-LAM against MoRe-CAM to show that LAM adds information beyond what CAM provides.","section":"Localization-informed Regularization, Eqs. (6), (7), (9)"},{"comment":"All results are single runs, and the hyperparameters (K, λ_h, λ_l, φ, d, τ, α, β, γ) are tuned on the validation split, so the reported margins (e.g., 11.4 mIoU drop without Lcre) lack variance estimates. Please provide mean and standard deviation over at least three seeds, or otherwise justify that the differences are larger than run-to-run noise.","section":"Table 3 (Ablation Studies)"}],"minor_comments":[{"comment":"The threshold values λ_h and λ_l are not specified in the main text; please give the actual values or point the reader to the appendix section where they are analyzed.","section":"Methodology, Eq. (6)"},{"comment":"The confusion ratio (FP/TP) is used as a key metric in Table 4 but is never formally defined; please define it explicitly and state how false positives and true positives are counted.","section":"Experimental Settings and Table 4"},{"comment":"The symbol T is used both for the multi-class token set and for the tail embeddings in the GCR module, which is confusing; please rename one of them (e.g., use T_tail for tails).","section":"Figure 2 and Methodology"},{"comment":"Please clarify which seed (MoRe-CAM or MoRe-LAM) is used to train the final segmentation model in Table 2, and whether the segmentation decoder is identical in both cases.","section":"Table 1 and Table 2"},{"comment":"The reference \"Ara. and Roth 2020\" is abbreviated and inconsistent with the other entries; please expand it to the full author list.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The pretraining mismatch is the decisive issue for the headline claim. If the authors can re-run ToCo and DuPL with ImageNet-21k pretraining and still show a clear gain, or demonstrate that pretraining accounts for only a small fraction of the gap, the paper would be substantially stronger. The LIR circularity concern is real but addressable with an additional control experiment. The paper's framing as the first to address the artifact issue may also need softening given prior work on registers and token contrast that is already cited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new part is the diagnosis of the artifact problem in LAM and the two regularization mechanisms: GCR, which treats class-patch attention as a directed graph with top-K neighbor selection and edge embeddings, and LIR, which uses CAM-derived confident and uncertain masks as contrastive supervision for class-patch consistency. The ablations in Table 3 are internally consistent and show each component contributes. The efficiency numbers are also favorable. This is honest, solid engineering on top of MCTformer/ToCo, with moderate novelty.\n\nThe soft spot is external validity. The paper uses ViT-B/16 pretrained on ImageNet-21k (Ridnik et al. 2021), while ToCo and DuPL—the key single-stage comparators—use ViT-B/16 pretrained on ImageNet-1k. The stress-test note is right: that difference alone can account for several points of mIoU. So the reported 76.4 vs 73.3 over DuPL, and the claim of beating multi-stage methods, does not establish that GCR+LIR are the decisive factors. The authors need a same-pretraining baseline, ideally their own method with 1k pretraining, or a re-run of ToCo/DuPL with 21k. The paper's internal ablations share the 21k backbone, so they show the modules help within that setting, but the SOTA headline is not supported as written.\n\nMinor issues: all results are single runs without error bars, hyperparameters are tuned on the validation split, and LIR explicitly supervises LAM with CAM-derived masks. That makes the LAM improvement partly inherited from CAM's biases; the paper does not quantify how much of the gain is due to the teacher signal versus the graph module. That is an interpretability issue, not a fatal one.\n\nWho is this for? Researchers working on ViT-based WSSS and class-patch attention. It deserves a serious referee, but the revision should address the pretraining confound, add seeds, and separate the LIR teacher effect. I would not cite it in its current form, but I'd want to see the revision.","headline":"MoRe makes a plausible case for regularizing class-patch attention in WSSS, but its SOTA claim is undercut by a pretraining mismatch with the key baselines.","tokens_in":15456,"tokens_out":4527,"would_cite":false,"duration_ms":35833,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MoRe claims that ViT class-patch attention artifacts in weakly supervised segmentation can be fixed by regularizing attention with a directed graph and CAM-guided losses.","keywords":["weakly supervised semantic segmentation","class activation maps","vision transformer","class-patch attention","attention regularization","localization attention maps","pseudo labels","semantic segmentation"],"falsifier":"Take a trained MoRe and retrain the LIR module with CAM-derived masks replaced by deliberately corrupted masks, e.g., masks whose confident regions are shifted off the true object or mixed with background, then measure LAM mIoU and the confusion ratio on PASCAL VOC val. If the artifact reduction persists despite wrong masks, the claimed mechanism is not doing the work; if the results track the corruption, the regularization is transferring CAM's biases rather than independently suppressing artifacts.","tokens_in":14365,"feed_emoji":"🎯","tokens_out":15474,"duration_ms":120202,"temperature":0.7,"pith_summary":"The paper reports a specific failure mode of Vision Transformer (ViT)-based weakly supervised semantic segmentation: the attention maps between class tokens and image patches, a popular alternative to Class Activation Maps (CAM), frequently light up semantically irrelevant patches. It argues that this 'artifact issue' has been neglected because previous work focused on oversmoothing among patch tokens rather than on the class-patch relation. MoRe adds two regularizers: a directed-graph module (GCR) that condenses only reliably related patch information into class tokens, and a CAM-guided module (LIR) that pulls class tokens toward confidently related and uncertain-but-relevant patches. With both, attention-based pseudo labels become competitive with multi-stage pipelines, reaching 76.4 mIoU on PASCAL VOC val and 47.4 mIoU on COCO val. A sympathetic reader would take the paper's core point to be that class-patch attention needs explicit regularization, and that this is a distinct and fixable problem from ViT oversmoothing.","feed_headline":"Regularized class-patch attention hits 76.4 mIoU on PASCAL VOC","feed_subtitle":"MoRe adds graph and CAM-informed losses to ViT attention, beating single- and multi-stage WSSS.","key_machinery":"The central object is the class-patch attention matrix, re-read as a directed graph to make its relation structure explicit. In GCR, each class token acts as a head $h_i$, each patch token as a tail $t_j$, and an edge embedding $e_{ij} = r_{ij} t_j + (1-r_{ij}) h_i$ parameterizes the directed relation, with only the top-$K$ tails retained as neighbors; the aggregation weight $S(h_i, e_{ij}, t_j) = \\mathrm{softmax}(t_j^T \\sigma(h_i + e_{ij}))$ decides which patch knowledge flows into class tokens, producing regularized class tokens $Q = \\delta_1(w_1(h_i + a_i)) + \\delta_2(w_2(a_i \\odot h_i))$. In LIR, CAM is thresholded (Eq. (6)) into confident ($M_c$) and uncertain ($M_u$) relation masks; $L_{\\mathrm{cre}}$ is a contrastive loss that pulls same-class class-patch pairs together and pushes different-class pairs apart, while $L_{\\mathrm{ure}}$ treats selected uncertain patches as positives to be pulled toward their class centroid. These modules carry the argument because LAM is the cosine-similarity score map between class and patch tokens, so regularizing that same score is a direct intervention on the localization signal.","core_discovery":"On the paper's own terms, MoRe establishes that the artifact issue in Localization Attention Maps (LAM) is caused by insufficient constraints on class-patch attention and can be resolved without architectural overhauls. The Graph Category Representation (GCR) module reinterprets class-patch attention as a directed graph whose nodes are heads (class side) and tails (patch side), with learnable edge embeddings and top-$K$ neighbor selection; the graph aggregation described by Eqs. (4)--(5) lets reliable patches enrich class tokens while excluding low-information artifact patches. The Localization-informed Regularization (LIR) module derives confident and uncertain relation masks from CAM via Eq. (6), then optimizes two losses: the confident relation enhancement loss $L_{\\mathrm{cre}}$ (Eq. (7)) contrasts same-class class-patch pairs against different-class pairs, and the uncertain relation enhancement loss $L_{\\mathrm{ure}}$ (Eq. (9)) pulls uncertain but relevant patches toward their class centroid. The combined objective Eq. (10) is trained end-to-end with a segmentation loss. The full system reports 76.4 mIoU on PASCAL VOC val, 75.0 on test, 47.4 on COCO val, pseudo seeds of 77.0 mIoU for LAM and 76.9 for CAM, and pseudo masks of 80.0/79.7 after DenseCRF.","pith_inferences":["Because LIR's supervision is only as good as CAM, the same two-module design should benefit from any stronger seed source; swapping in better CAM or iterative refinement would be a direct test and a likely improvement.","The paper frames artifacts as a class-patch relation problem distinct from patch oversmoothing, which predicts that MoRe-style regularization should transfer to other ViT localization tasks, such as weakly supervised object detection or open-vocabulary segmentation, where class tokens are read out as spatial maps.","The directed head/tail formulation implies that the asymmetry of class-to-patch vs patch-to-class information matters; a testable prediction is that symmetric or bidirectional attention regularization would capture less of the artifact suppression gain.","Current evidence is on PASCAL VOC and COCO with balanced classes; a natural stress test is class-imbalanced or fine-grained datasets, where CAM is noisier and the confidence masks in Eq. (6) are more likely to be wrong."],"forward_implications":["Single-stage training suffices: MoRe reaches 76.4 mIoU on PASCAL VOC val and 47.4 mIoU on COCO val, surpassing recent multi-stage systems as well as single-stage ones.","Regularized LAM can act as a high-quality pseudo-label source: MoRe produces pseudo seeds around 77.0 mIoU and pseudo masks of 80.0/79.7 mIoU with DenseCRF.","The two regularizers are complementary, not redundant: the ablation shows removing the confident relation loss drops mIoU by 11.4 points, removing the uncertain relation loss lowers recall from 88.0 to 85.7, and removing the graph module drops mIoU by 4.3 points.","End-to-end efficiency is part of the claim: the full pipeline takes 372 minutes and 12.1 GB GPU memory, cheaper than the multi-stage LAM baseline and the single-stage competitors."],"supporting_citations":[{"why":"Supplies the multi-class token ViT setting that produces class-specific LAM, the backbone MoRe inherits.","marker":"Xu et al. 2022"},{"why":"Provides the LAM-based SOTA baseline and the CAM-LAM fusion strategy that MoRe extends and compares against.","marker":"Xu et al. 2024a"},{"why":"First extracted LAM from class-patch attention, establishing the attention map as the localization signal under study.","marker":"Gao et al. 2021"},{"why":"Supplies evidence that ViT aggregates global semantics in low-information patches, motivating the artifact diagnosis.","marker":"Darcet et al. 2023"},{"why":"Supports the claim that unrelated patches frequently correlate with class tokens during attention, causing false activations.","marker":"Sun et al. 2024"},{"why":"Provides the PTC loss for oversmoothing and the ToCo single-stage baseline; shows oversmoothing regularization alone does not remove artifacts.","marker":"Ru et al. 2023"},{"why":"Provides the DuPL single-stage SOTA baseline and training settings that MoRe surpasses.","marker":"Wu et al. 2024"}],"fun_headline_variants":["MoRe fixes attention artifacts, hits 76.4 mIoU on PASCAL VOC","Graph-based attention regularization for WSSS, top on PASCAL VOC","MoRe: regularized class-patch attention beats multi-stage WSSS","Attention artifact fix: MoRe reaches 76.4 mIoU on PASCAL VOC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach rests on trusting the class activation maps to know, per image, which regions are confidently part of an object and which regions are uncertain; if those maps are wrong about an object's extent, the contrastive losses will bake that error into the attention maps and the improvements will partly just copy CAM's biases.","fun_headline_variants_meta":{"raw":{"variants":["MoRe fixes attention artifacts, hits 76.4 mIoU on PASCAL VOC","Graph-based attention regularization for WSSS, top on PASCAL VOC","MoRe: regularized class-patch attention beats multi-stage WSSS","Attention artifact fix: MoRe reaches 76.4 mIoU on PASCAL VOC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000803,"raw_usage":{"total_tokens":3618,"prompt_tokens":1125,"completion_tokens":2493,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":741,"completion_tokens_details":{"reasoning_tokens":2402}},"tokens_in":741,"tokens_out":2493,"duration_ms":16003,"temperature":1.0,"reasoning_tokens":2402,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:19:11.275588+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained MoRe and retrain the LIR module with CAM-derived masks replaced by deliberately corrupted masks, e.g., masks whose confident regions are shifted off the true object or mixed with background, then measure LAM mIoU and the confusion ratio on PASCAL VOC val. If the artifact reduction persists despite wrong masks, the claimed mechanism is not doing the work; if the results track the corruption, the regularization is transferring CAM's biases rather than independently suppressing artifacts.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"First extracted LAM from class-patch attention, establishing the attention map as the localization signal under study."}],"review_version":1}