{"id":"4702962e-cef9-45af-a479-8c730a0c0c6e","arxiv_id":"2412.18273","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"SBV improves open-vocabulary detection by adaptively sampling semantic concept windows, switching between global, middle, and local views, and masking CLIP attention, gaining 2.6 AP50 on COCO and 0.5 mAP on LVIS over BARON.","lead":"This paper proposes SBV, a method that samples bags of semantically related views to improve open-vocabulary object detection. It reports small but consistent gains over the prior BARON method while claiming an 80% reduction in CLIP compute.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 80.3% CLIP FLOPs reduction is computed with an invalid sparse-attention assumption; standard masked attention does not reduce FLOPs, so the central efficiency claim is unsupported without runtime measurements.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: the 80.3% FLOPs reduction depends on the assumption that attention masks reduce FLOPs proportionally to unmasked positions, which is not how standard masked attention works. My reading of Sec. A.5 and Eq. (9) confirms this: the formula counts only the unmasked positions in the attention FLOPs, but no sparse attention implementation is described. The paper also provides no measured runtime, so the efficiency claim rests entirely on this unvalidated model. This is the most load-bearing issue because the abstract's central claim is a combination of accuracy and efficiency; while the 2.6 AP50 / 0.5 AP gains are modest and hyperparameters are tuned on the same validation split (Sec. B.2), the efficiency number is both specific and likely incorrect. The reader's CONDITIONAL verdict is appropriate: the core idea is plausible, but the strongest claim requires verification. My concern does not move the verdict, as the reader already flagged this as the weakest assumption; hence UNCHANGED.","tokens_in":19176,"tokens_out":6588,"duration_ms":59140,"concrete_test":"Profile the CLIP image encoder's actual forward-pass wall-clock time and FLOPs (e.g., with torch.profiler) on a representative batch of the region crops used in training, both with and without the noise mask, and compare the measured ratio to the prediction of Eq. (9) (FLOPs proportional to N/W). Then recompute the PFLOPs values in Tab. 2 using a standard dense FLOPs counter that does not assume masked positions are skipped. If the measured runtime reduction is far smaller than predicted, or if the recomputed FLOPs for the noise-mask-only row do not drop by more than half relative to BARON, the 80.3% reduction claim is unsupported and should be revised to reflect only the region-count savings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central efficiency claim (80.3% CLIP FLOPs reduction vs BARON) is supported only by the FLOPs model in Sec. A.5, specifically Eq. (9): FLOPs_SA = 2BHD × WN, where N is the number of unmasked attention positions. This formula assumes that masking attention entries reduces the FLOPs of the QK^T and attention-value multiplications proportionally to the number of unmasked positions. In standard implementations (such as PyTorch's scaled_dot_product_attention or typical CLIP code), attention masks are applied by setting masked logits to -inf before softmax; the QK^T and value-weighting matrix multiplications still execute at full size over all W positions. The paper does not mention using sparse attention kernels or any implementation that would skip masked entries. Consequently, the claimed savings from the noise mask (e.g., 55.3 → 25.7 PFLOPs in Tab. 2 for adding the noise mask alone) are likely artifacts of the model rather than real reductions. Even the overall region-count reduction (216 regions for BARON vs 56 for SBV, Tab. 7) accounts for at most a 74% FLOPs reduction if FLOPs scale linearly with region count; the additional 80.3% figure depends precisely on the unvalidated mask savings. Because the paper reports no wall-clock time or profiler-based FLOPs, the efficiency claim is not empirically grounded. This is the most load-bearing concern because the abstract's headline contribution combines a 2.6 AP50 / 0.5 AP accuracy gain with a dramatic efficiency gain; if the efficiency number is wrong, the central claim is materially overstated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SBV (Sampling Bag of Views), an open-vocabulary object detection method built on the BARON bag-of-regions framework. SBV first constructs a graph-based 'canvas' over RPN proposals and probabilistically samples a bag of semantically related 'concepts' through edge traversal. It then selects among three hierarchical views (global, middle, local) for each concept using a representation-switching rule, and uses noise masks and view masks inside the CLIP image encoder to suppress background patches and emphasize important views. Experiments on OV-COCO and OV-LVIS report a novel-category box AP50 of 36.6 and a rare-category mask mAP of 23.1, respectively, surpassing prior state-of-the-art methods, and the paper claims an 80.3% reduction in CLIP FLOPs compared to BARON.","tokens_in":19543,"tokens_out":5042,"duration_ms":47600,"significance":"If the reported numbers are reliable, SBV would be a useful contribution to open-vocabulary detection: it advances the state of the art on two widely used benchmarks and introduces a more semantic sampling strategy than BARON's spatial neighbors. The paper includes a comparatively detailed description of the adaptive sampling procedure, ablations over its main components, and qualitative visualizations that help the reader understand the behavior of the canvas and edge-generation modules. However, the main empirical gains are modest (2.6 AP50 on OV-COCO, 0.5 mAP on OV-LVIS over BARON), the efficiency claim is based on a FLOPs model that appears to be invalid for standard masked attention, and the system has many hyperparameters tuned on the same validation set used for reporting. The central efficiency claim is therefore not yet supported, which limits the paper's current significance.","major_comments":[{"comment":"The claimed 80.3% reduction in CLIP FLOPs rests on Eq. (9), which models self-attention cost as proportional to W·N, where N is the number of unmasked positions. In standard dense attention implementations, an attention mask is applied by adding -inf or 0 to the logits before softmax; the QK^T multiplication and the attention-value multiplication are still performed over all W positions. The paper does not state that it uses sparse attention kernels, and no wall-clock timing or profiler-based FLOPs measurements are reported anywhere in the main text or appendix. Consequently, the FLOPs numbers in Table 2 and the abstract's efficiency claim are unsupported as stated.","section":"A.5, Eq. (9)"},{"comment":"The implementation of the noise mask is described in Eq. (4) as adding -inf to the logits and then multiplying the softmax output by a view mask. This operates on already-computed attention weights and does not reduce the arithmetic cost of the underlying matrix multiplications in a dense implementation. The ablation rows in Table 2 that attribute large PFLOPs reductions to the noise mask alone (55.3 to 25.7) therefore appear to be artifacts of the same incorrect FLOPs model rather than real computational savings. Please either provide measurements from an actual sparse-attention implementation or revise the efficiency claims.","section":"4.2.2, Eq. (4) and Table 2"},{"comment":"The hyperparameters η, N, τ, view weights, and the scaling factor s are selected by searching on the COCO validation split, and the same split's AP numbers are then reported as the final results in Tables 1–7. Because the paper reports single-run results without variance or significance estimates, the gains of 2.6 AP50 over BARON on OV-COCO and 0.5 mAP on OV-LVIS could reflect validation-set selection rather than a robust improvement. Please report results with a fixed default hyperparameter configuration, or provide multiple-seed statistics, to support the state-of-the-art claim.","section":"5.2 and B.2"}],"minor_comments":[{"comment":"The text states that the noise mask affects both self-attention and MLP FLOPs, but Eq. (10) for the MLP layer contains no dependence on the mask N. Please clarify which components actually change.","section":"A.5, Eq. (10)"},{"comment":"The symbol N is used both for the noise mask and for the number of unmasked positions in the FLOPs formula; using different symbols would avoid confusion.","section":"Eq. (3)-(4) and Eq. (9)"},{"comment":"A large part of the claimed efficiency gain is already explained by the reduction in the number of regions per bag (216 for BARON versus 56 for SBV). The mask-based savings should be separated from the sampling-based savings so the reader can understand the sources of the reduction.","section":"Table 7"},{"comment":"The qualitative comparisons in Figures 5 and 6 show only a few favorable examples; including a broader set or a quantitative explainability metric would strengthen the claim that SBV focuses on meaningful concepts.","section":"5.3"},{"comment":"The hyperparameter guidance in the supplementary is helpful, but the main text should include a compact version of Table 8, since several of these choices (e.g., number of edges E, interval Δ, view weights) are not obvious from the equations alone.","section":"B.2"}],"recommendation":"major_revision","confidential_remarks":"The FLOPs issue is the key blocker. The current 80.3% figure is derived from a masked-attention model that is not valid for standard dense attention, and the paper provides no runtime data. This is fixable—the authors could report wall-clock time, use a genuinely sparse kernel, or remove the efficiency claim—but as submitted the central efficiency contribution is unsupported. The validation-set hyperparameter tuning and single-run numbers are a further concern that should be addressed with clearer reporting. The performance gains are modest but positive, and the adaptive sampling idea is interesting; with proper efficiency evidence the paper could be a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: SBV is a sensible extension of BARON with genuinely new machinery—canvas-based semantic sampling, edge generation, representation switching, and view/noise attention masks—and the accuracy gains on OV-COCO look credible. The bigger problem is the headline efficiency claim. The 80.3% CLIP FLOPs reduction is computed from Eq. (9), which assumes masked attention skips work proportionally to unmasked positions. Standard CLIP implementations only zero softmax entries; QK^T and attention-value matmuls still run at full size. The paper gives no sparse kernel, no profiler output, and no wall-clock time, so the PFLOPs column in Tab. 2 is an artifact of the FLOPs model rather than a measured saving. That is load-bearing: the abstract sells a modest accuracy gain plus a dramatic efficiency gain, and the efficiency half is unsupported.\n\nWhat is actually new and good: the canvas-and-edge sampling is a real departure from BARON's fixed nearest-neighbor bag, and the ablations in Tabs. 2, 3, 4, 6, and 7 support the contribution of each component. The 2.6 AP50 improvement on novel COCO categories is meaningful, though the 0.5 mask AP gain on LVIS is small enough to be fragile. The paper is also candid in the supplement about hyperparameter choices, which helps.\n\nSoft spots, in proportion: the FLOPs issue is the worst and should be the first thing a referee checks. The accuracy numbers are single-run with no error bars. Several hyperparameters (η, N, τ, view weights) appear to be tuned on the same validation split used for the headline numbers; that is a selection effect, and the paper should report it directly and ideally use held-out tuning. The noise embeddings are learned inside the same training pipeline, which is a mild self-reference but not a fatal one.\n\nWho gets value: people working on open-vocabulary detection and distillation from frozen VLMs. The sampling idea is worth knowing even if the efficiency story collapses. My recommendation: send it to peer review, but condition acceptance on measured runtime or profiler-based FLOPs, multiple seeds, and a transparent account of validation-based tuning. If the efficiency claim does not survive, the contribution shrinks to a modest accuracy improvement over BARON—still publishable, but with a different headline.","headline":"Solid extension of BARON with genuinely new sampling and view-mask machinery, but the 80.3% FLOPs headline rests on an unvalidated sparse-attention assumption and needs runtime verification.","tokens_in":20092,"tokens_out":2310,"would_cite":false,"duration_ms":22803,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sampling concept views beats region bags for open-vocabulary detection.","keywords":["open-vocabulary object detection","vision-language models","bag of views","adaptive sampling","compositional structure","CLIP distillation","region proposal network","noise masks"],"falsifier":"Measure actual FLOPs or wall-clock time of CLIP inference with and without the noise and view masks using a standard implementation that accounts for dense matrix multiplications; if masked positions do not reduce the dimensions of the QK^T and attention-value products, the 80.3% figure will not reproduce.","tokens_in":18959,"feed_emoji":"🎯","tokens_out":5714,"duration_ms":46072,"temperature":0.7,"pith_summary":"Open-vocabulary detection asks a detector trained on base categories to recognize novel categories by borrowing knowledge from a frozen vision-language model. This paper argues that the way the detector samples context around a proposal matters more than the alignment loss itself. It proposes SBV: instead of grabbing spatially nearest regions, which often include background, it builds a canvas of candidate concepts, draws probabilistic edges between proposals to find semantically related concepts, and groups them into a bag. Each concept in the bag is then viewed at global, middle, and local scales, with masks that suppress background and weight important views. SBV reports 36.6 AP50 on novel COCO categories and 23.1 mask mAP on novel LVIS categories, and it claims an 80.3% reduction in CLIP FLOPs versus BARON.","feed_headline":"Concept-based views beat region bags in open-vocabulary detection","feed_subtitle":"New sampling strategy lifts novel-category AP by 2.6 on COCO and cuts reported CLIP FLOPs by 80.3%.","key_machinery":"The central object is the bag of views, built from concept windows. A concept window is a crop that merges a region proposal with a nearby visual concept chosen probabilistically from a canvas of RPN boxes; the bag is the set of these merged windows. This object carries the argument in three ways: it samples only windows that contain at least one semantic concept, so fewer CLIP inputs are needed; it switches each concept to a global, middle, or local view at an optimal scale via a threshold on size ratio and co-occurring concept count; and it applies noise masks and view masks inside CLIP's self-attention to suppress background and weight important patches. The paper also adds extra farthest RPN proposals to generate more edges when the RPN finds too few proposals, which prevents the method from missing concepts in images with sparse detections.","core_discovery":"The central claim is that compositional structure in vision-language models is better exploited by sampling semantically related concepts around each region proposal and aligning a bag of views, rather than a bag of spatially neighboring regions. The paper demonstrates this by replacing BARON's fixed-window neighbor sampling with an adaptive sampling strategy: RPN boxes are reduced, a canvas records probabilities of nearby concepts, edges are generated between proposal pairs, and a distance-and-aspect-ratio weighted selection picks representative concepts. For each concept, representation switching chooses among global, middle, and local views based on size ratio and co-occurring concept count; noise masks and view masks are applied inside CLIP's self-attention. Combined with Faster R-CNN, SBV outperforms BARON by 2.6 box AP50 on OV-COCO novel categories and 0.5 mask mAP on OV-LVIS rare categories, while reporting an 80.3% reduction in CLIP FLOPs.","pith_inferences":["Inference (editorial): the canvas-and-edge concept selection is not tied to CLIP; it could be reused for other region-based alignment or grounding objectives.","Inference (editorial): the 80.3% FLOPs reduction should be verified on real hardware, because attention masks that only zero softmax entries do not skip the underlying matrix multiplications in many implementations.","Inference (editorial): representation switching suggests the optimal context size depends on scene composition, so the threshold could be learned from data rather than fixed.","Inference (editorial): since the sampler relies only on RPN boxes and objectness scores, it may extend to weakly supervised or self-supervised detection settings without ground-truth boxes."],"forward_implications":["On OV-COCO, novel-category box AP50 rises from 34.0 for BARON to 36.6; on OV-LVIS, rare-category mask mAP rises from 22.6 to 23.1.","SBV samples 56 regions per image instead of BARON's 216, so fewer CLIP forward passes are needed for the same alignment objective.","The noise mask and view mask concentrate CLIP attention on foreground patches, which is the source of the reported 80.3% FLOPs reduction.","The gains come from fewer, semantically valid CLIP inputs rather than larger models or extra training data, since the backbone and schedule match BARON."],"supporting_citations":[{"why":"Supplies the BARON baseline that SBV replaces and improves, and the bag-of-regions representation.","marker":"[25]"},{"why":"Provides RPN boxes and objectness scores used to build the canvas and candidate concepts.","marker":"[22]"},{"why":"Supplies the frozen CLIP image and text encoders whose embeddings are aligned.","marker":"[20]"},{"why":"Shows that VLMs capture compositional structures in complex scenes, motivating the concept-bag approach.","marker":"[3]"},{"why":"Defines the ViLD distillation baseline and the hand-crafted prompts used on OV-COCO.","marker":"[8]"},{"why":"Provides the COCO dataset for the unnecessary-neighbor analysis and the OV-COCO benchmark.","marker":"[15]"},{"why":"Provides the LVIS dataset and the rare-category split for the OV-LVIS benchmark.","marker":"[9]"},{"why":"Defines the base/novel category split used for OV-COCO evaluation.","marker":"[29]"}],"fun_headline_variants":["Concept bags beat region bags in open-vocab detection","Sampling concept views cuts CLIP compute by 80.3%","Open-vocab detector uses adaptive concept bags for gains","Adaptive concept sampling lifts open-vocab AP by 2.6","Concept-bag views reduce CLIP FLOPs 80% in OVD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed 80.3% FLOPs reduction relies on the assumption that masking positions in CLIP's self-attention removes those positions from the FLOP count; if the underlying matrix multiplications still run at full size, the reported efficiency gain is not realized.","fun_headline_variants_meta":{"raw":{"variants":["Concept bags beat region bags in open-vocab detection","Sampling concept views cuts CLIP compute by 80.3%","Open-vocab detector uses adaptive concept bags for gains","Adaptive concept sampling lifts open-vocab AP by 2.6","Concept-bag views reduce CLIP FLOPs 80% in OVD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1512,"prompt_tokens":965,"completion_tokens":547,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":454}},"tokens_in":581,"tokens_out":547,"duration_ms":4605,"temperature":1.0,"reasoning_tokens":454,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:50:54.205865+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure actual FLOPs or wall-clock time of CLIP inference with and without the noise and view masks using a standard implementation that accounts for dense matrix multiplications; if masked positions do not reduce the dimensions of the QK^T and attention-value products, the 80.3% figure will not reproduce.","supporting_citations":[{"cited_title":"Aligning bag of regions for open- vocabulary object detection","cited_arxiv_id":null,"evidence_quote":"Supplies the BARON baseline that SBV replaces and improves, and the bag-of-regions representation."},{"cited_title":"Girshick, and Jian Sun","cited_arxiv_id":null,"evidence_quote":"Provides RPN boxes and objectness scores used to build the canvas and candidate concepts."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen CLIP image and text encoders whose embeddings are aligned."},{"cited_title":"Maskclip: Masked self-distillation advances contrastive language-image pretraining","cited_arxiv_id":null,"evidence_quote":"Shows that VLMs capture compositional structures in complex scenes, motivating the concept-bag approach."},{"cited_title":"Open-vocabulary object detection via vision and language knowledge distillation","cited_arxiv_id":null,"evidence_quote":"Defines the ViLD distillation baseline and the hand-crafted prompts used on OV-COCO."},{"cited_title":"Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C","cited_arxiv_id":null,"evidence_quote":"Provides the COCO dataset for the unnecessary-neighbor analysis and the OV-COCO benchmark."},{"cited_title":"Girshick","cited_arxiv_id":null,"evidence_quote":"Provides the LVIS dataset and the rare-category split for the OV-LVIS benchmark."},{"cited_title":"Open-vocabulary object detection using captions","cited_arxiv_id":null,"evidence_quote":"Defines the base/novel category split used for OV-COCO evaluation."}],"review_version":1}