{"id":"3a081646-5beb-447f-bf1e-1832df800010","arxiv_id":"2506.21233","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ReME builds a cleaned, synonym-enriched segment-text reference set from real images and shows that simple similarity retrieval on it beats 14 prior training-free open-vocabulary segmentation methods across ten benchmarks.","lead":"This paper shows that a carefully cleaned library of image segment and text pairs lets a simple matching step outperform far more complex training-free open-vocabulary segmentation methods on ten benchmarks. The result points to data quality, not model adaptation or retrieval complexity, as the main performance lever in this setting.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim hinges on group-based filtering (Sec 3.2) correctly treating misaligned pairs as intra-modal outliers; this is unvalidated for polysemous or multi-modal root groups, and the filter drives the largest gain (Table 2).","rationale":"The paper's central claim is that a high-quality real-image reference set, cleaned by intra-modal filtering and synonym enrichment, is what enables a simple retrieval to beat all 14 training-free baselines, and that data quality rather than retrieval design drives the gain. Table 2 isolates the contributors and shows group-based filtering is the dominant one (70.03 to 91.10 on VOC-20). The filter's validity rests entirely on the assumption that correct segments for a root noun form a coherent, roughly unimodal cluster so that misaligned pairs are outliers. This assumption is not tested quantitatively. String-matched root nouns are a known source of multi-modal groups (polysemy), and even cohesive categories have wide intra-class variance; the median-based filter with a fixed 30% drop ratio can then remove valid references instead of (or in addition to) misaligned ones. The qualitative UMAPs and the comparison with CLIP-score filtering are suggestive but do not establish that the filter is detecting misalignment rather than simply discarding a fraction of the distribution. A controlled experiment with injected misalignment and known ground truth would settle this directly. If the filter is in fact deleting valid hard examples, the 'data quality' narrative weakens: the gain could reflect an easier reference distribution, not better alignment, and the approach may generalize less well to long-tail or out-of-distribution queries. I agree with the reader's weakest_assumption. The verdict should remain CONDITIONAL: the concern is specific and testable, and the paper's other evidence (data robustness across image sources, transferability to FreeDA/CAT-Seg, consistent gains on ten benchmarks) prevents it from being fatal, but the open question blocks full ACCEPT.","tokens_in":26121,"tokens_out":10686,"duration_ms":130898,"concrete_test":"Use COCO Stuff training GT masks and labels to create a reference set of known-correct segment-text pairs; randomly reassign labels on a fixed fraction (e.g., 30%) of segments to inject known misalignment. Run the Sec 3.2 group-based filter with δ_filter=30% and measure precision/recall of detecting the injected errors. Also compute the fraction of root groups whose similarity-to-median distribution is multimodal (e.g., Hartigan dip test or Gaussian-mixture BIC). High precision/recall and few multimodal groups would support the assumption; near-chance detection or many multimodal groups would show the filter is deleting valid data, weakening the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is group-based filtering (Sec 3.2): Table 2 shows it lifts VOC-20 mIoU from 70.03 to 91.10, the largest component of the reported gain. The mechanism assumes that segments sharing the same root noun have inherently consistent visual features, so misaligned pairs appear as outliers. This unimodality assumption is not guaranteed. Root nouns are matched by string, not sense: 'bat', 'mouse', 'crane', and 'seal' each group multiple visual concepts, and even single concepts such as 'chair' or 'dog' contain large intra-class variation. In a multi-modal group the median center can fall between modes, so the 30% lowest-similarity criterion may delete an entire valid subpopulation rather than misaligned pairs. The paper's evidence is qualitative (Fig 3, Fig A4) plus the Table 3 contrast with CLIP-score filtering; no precision/recall measure of the filter against known misalignment exists, and no analysis of how many root groups are multimodal. Supplementary D acknowledges the drop-instead-of-correct choice but not the risk of dropping correct-but-unusual pairs. If the filter improves scores by pruning hard examples rather than by correcting alignment, the central claim that data quality (alignment), not retrieval strategy, drives the gains is weakened, and the benefit may not transfer to open-vocabulary settings beyond the benchmark distribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ReME, a training-free open-vocabulary segmentation framework that constructs a reference set of segment-text pairs from real images (COCO-2017) using LLaVA-generated descriptions and CLIP-based pairing, then enhances data quality through two steps: group-based intra-modal filtering (dropping the lowest-similarity segments within each root-noun group) and synonym-based semantic enrichment. At inference, a simple similarity-based retrieval over this reference set assigns labels to class-agnostic test segments. The paper reports state-of-the-art results over 14 training-free OVS baselines on ten benchmarks, with ablations showing that group-based filtering contributes the largest performance gain (VOC-20 mIoU from 70.03 to 91.10), and additional experiments on data robustness, hyperparameters, backbone choices, description generators, and segmenters.","tokens_in":26377,"tokens_out":7495,"duration_ms":83860,"significance":"If the claims hold, this is a meaningful contribution: it demonstrates that data quality, rather than retrieval complexity or model adaptation, can be the main lever for training-free OVS, and it does so with broad empirical coverage (ten benchmarks, 14 baselines, component ablations, hyperparameters tuned on a held-out 1k-image subset, an explicit limitations paragraph, and released code). The same-modal retrieval design (visual-to-visual and text-to-text similarities) is elegant and avoids the known weakness of cross-modal CLIP scores. The main risks are that the largest pipeline component rests on an unquantified unimodality assumption, and that one data-cleaning step is described only in the supplement and excluded from the main ablation.","major_comments":[{"comment":"The group-based filtering step assumes that segments grouped by the same root noun form a visually unimodal cluster, so that misaligned pairs appear as outliers. This assumption is load-bearing because Table 2 attributes the largest single gain to this step (VOC-20: 70.03 to 91.10). The manuscript provides only qualitative UMAP evidence (Fig. 3, Fig. A4) and does not quantify how many root groups are multi-modal or whether the filter's precision/recall against known misalignments is high. Please add a per-group analysis (e.g., distribution of within-group similarities, detected modes) and evaluate the filter on a subset with known ground-truth alignment (e.g., COCO Stuff segments with GT class labels). Without this, the improvement could stem from removing hard but correctly-paired segments rather than from correcting misalignment, which would weaken the central data-quality claim.","section":"Sec. 3.2, Tables 2 and 3"},{"comment":"The pipeline described in Sec. 3.2 omits the 'Filtering Ambiguous Labels' step, which is only presented in Supplementary B.2. This step removes label roots whose group sizes exceed an 'inflection point' (e.g., 'background', 'scene', 'atmosphere', 'image'), and it appears to be applied before or within the data-enhancing phase. If Table 2's 'Base set (no enhancement)' already includes this filtering, then the ablation does not isolate the contribution of group-based filtering as claimed. Moreover, the inflection point is an additional free parameter that is not part of the grid search reported in Sec. 4.3. Please state exactly which preprocessing steps are included in the base set, ablate the ambiguous-label filter separately, and report how the inflection threshold is selected.","section":"Supplementary B.2 vs. Sec. 3.2, Table 2"},{"comment":"For the rows 'ReME(Ours - VOC)' and 'ReME(Ours - ADE)', the manuscript does not specify which splits of VOC and ADE are used to construct the reference set. If the validation splits of these datasets (the same ones used for evaluation) are included, the robustness comparison would be affected by evaluation leakage. Please specify the exact image splits used and, if necessary, rerun with train-only images.","section":"Table 1, data robustness rows"},{"comment":"Because LLaVA description generation is stochastic and the reference set construction involves sampling, the reported single-run numbers may vary. This matters for benchmarks where the margin over the second-best method is small (e.g., A-847: 8.4 vs 6.6; PC-459: 14.1 vs 12.0). Please report results over at least three independent runs (or show that decoding is deterministic with a fixed seed) and provide mean and standard deviation for the main comparison tables.","section":"Sec. 4.1 and all result tables"}],"minor_comments":[{"comment":"The inference-time comparison says 'All experiments are conducted on two NVIDIA 4090 GPUs', but it is unclear whether the reported seconds/image use both GPUs and how the per-image average is computed; please clarify the measurement protocol.","section":"Table 8"},{"comment":"The supplementary tables use the header 'A VG 10', which appears to be a typo for 'AVG10' (average across ten benchmarks); please define this metric in the main text or table captions.","section":"Supplementary Tables A4-A9"},{"comment":"The sentence 'Across 10 benchmarks, ReME achieves the highest mIoU in 6 benchmarks with VOC and 7 with ADE' should specify that these wins are among SAM-free baselines, since the table also includes SAM-involved methods and the bolding convention applies only to the SAM-free comparison.","section":"Sec. 4.2"},{"comment":"The binary matrix O_ref represents segment-label associations, but after synonym enrichment a segment can have multiple labels; please clarify that O_ref entries remain binary and that each synonym-enriched phrase is treated as a separate label column.","section":"Eqs. (1)-(3)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has strong empirical breadth and a plausible central claim, but the largest pipeline component (group-based filtering) rests on an unquantified unimodality assumption, and a data-cleaning step appears only in the supplement and is not ablated in the main paper. These are fixable with additional experiments and clearer reporting. I also note that the explicit limitations paragraph is a positive signal, but it does not address the multimodal-group risk. My recommendation is major revision rather than rejection because the core idea is sound and the requested validations are within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a real result, not a packaging job. The paper's central claim is that a reference set of real segment-text pairs, cleaned by intra-modal group-based filtering and synonym enrichment, makes a simple similarity retrieval beat 14 training-free baselines on ten benchmarks. The ablations back the claim: Table 2 shows the base set at 70.03 mIoU on VOC-20, group filtering alone to 91.10, and both components to 92.34. Table 3 is the key contrast—group-based filtering with intra-modal DINOv2 scores (92.34) versus global or group-based CLIP-score filtering (79.34/80.05). That is a decisive difference, and unlike many papers the authors isolate the mechanism. They also tune both hyperparameters on a 1k-image subset of the training split, not on the eval sets, and they disclose the backbone-selection rule for baselines instead of hiding it. The supplementary transferability experiments (Table A2), where the ReME reference set improves CAT-Seg and FreeDA, are a strong piece of evidence that the data itself carries the value.\n\nThe soft spots are real but proportionate. First, no error bars or repeated runs anywhere, even though LLaVA decoding is stochastic; Table A9 in the supplement does report mean±std over three GPT4o prompting runs for free-form queries, but the main benchmarks have no variance. Second, baseline numbers are taken from prior papers without re-running; the disclosed favorable backbone rule helps, but it is not a substitute for a head-to-head rerun. Third, the load-bearing group-filtering step assumes each root-noun group is visually unimodal. For polysemous roots like 'bat' or 'mouse', or high-variance classes like 'chair', a 30% drop could delete a valid subpopulation rather than misaligned pairs. The paper's evidence for the mechanism is qualitative (Fig 3, Fig A4) plus the Table 3 contrast; I would have liked a precision/recall test against known misalignment. The limitation paragraph in Supplementary D honestly acknowledges the drop-instead-of-correct choice, which mitigates the concern.\n\nWho this is for: anyone working on open-vocabulary segmentation or data-centric methods for dense prediction. It deserves a serious referee; the central empirical claim is strong enough to warrant a full review. I'd ask the authors for error bars, a direct evaluation of the filter, and ideally a rerun of the closest baselines.","headline":"Solid data-centric OVS paper: the intra-modal filtering result holds up, the ablations are honest, and the main soft spots (error bars, filter precision) are fixable rather than fatal.","tokens_in":26944,"tokens_out":2508,"would_cite":true,"duration_ms":27355,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Data quality, not model surgery, decides training-free open-vocabulary segmentation.","keywords":["open-vocabulary semantic segmentation","training-free","data-centric","reference set construction","intra-modal filtering","CLIP","DINOv2","similarity retrieval"],"falsifier":"Have humans check the segments that ReME's group-based filter drops on visually heterogeneous classes (e.g., 'furniture', 'vehicle', 'bird'): if many dropped pairs prove correctly labeled, the filter is removing good data, not misalignments. A quantitative version: replace the intra-modal filter with an oracle built from ground-truth segment-text labels; if the oracle does not reproduce most of the filtering gain (91.10 vs 70.03 on VOC-20), then the filtering mechanism claimed to carry the result is not the real driver.","tokens_in":25853,"feed_emoji":"🧹","tokens_out":8795,"duration_ms":94836,"temperature":0.7,"pith_summary":"This paper argues that training-free open-vocabulary segmentation (OVS) is bottlenecked by the quality of the reference data, not by the cleverness of the segmentation or retrieval machinery. To show this, the authors construct a reference set of segment-text pairs from real COCO-2017 images, using a vision-language model to pair segments with noun-phrase labels, then cleaning the pairs by filtering outliers within each label group using same-modality (visual) similarity, and finally enriching labels with synonyms found among text embeddings. With only a simple similarity retrieval, this cleaned set outperforms all 14 prior training-free OVS methods across ten benchmarks, and on VOC-20 it lifts mIoU from 70.03 for the raw base set to 92.34. If the paper is right, then the practical lever for improving training-free dense vision understanding is data curation, building well-aligned, semantically rich reference sets, rather than further post-hoc model adaptation.","feed_headline":"Better data beats better tricks for open-vocabulary segmentation","feed_subtitle":"Clean real-image segment-text pairs let simple retrieval beat all 14 prior training-free methods on ten benchmarks.","key_machinery":"The load-bearing object is the reference set of segment-text pairs, refined by two intra-modal operations. Group-based filtering gathers segments whose labels share a root noun, computes a median visual feature for each group, and drops the configured fraction of segments farthest from that median; semantic enriching then adds synonym labels discovered by pairwise text-embedding similarity among root nouns, e.g., adding 'a small kitten' to a segment labeled 'a small cat'. Retrieval is a two-hop softmax: test segments vote for reference labels, reference labels vote for the given test classes, and the composed votes are summed over each test segment's mask to produce pixel-level predictions. The essential mechanism is collective consistency, since segments that share a label should look alike, mislabeled pairs become detectable outliers in the visual feature space.","core_discovery":"The paper's central claim is that for training-free open-vocabulary segmentation, the quality of the external reference set, not the sophistication of the retrieval or attention mechanism, determines performance. ReME builds that reference set from real COCO-2017 images alone: class-agnostic segments are paired with noun-phrase labels using CLIP, misaligned pairs are removed by comparing each segment's visual embedding with the median embedding of its root-noun group, and labels are diversified by adding synonym noun phrases discovered from text-embedding similarity. At test time, a simple softmax-weighted similarity retrieval assigns class probabilities to each segment and aggregates them through the class-agnostic masks. On VOC-20 the raw base set scores 70.03 mIoU, group-based filtering alone raises this to 91.10, and the full pipeline reaches 92.34, outperforming all 14 prior training-free OVS methods across ten benchmarks.","pith_inferences":["The same group-then-outlier filtering idea could be applied to other multimodal resources, such as image-text or video-caption datasets, to prune misaligned samples before training; the paper's mechanism is not specific to segmentation.","Because the largest gain comes from filtering, splitting polysemous root nouns (e.g., 'bat' as animal vs. equipment) or using per-group adaptive drop ratios, as the supplementary weighted variant already explores, should further improve robustness on heterogeneous categories.","If data quality is the binding constraint, scaling the real-image corpus beyond COCO-2017 or using even richer captioners should keep lifting performance, and the reference set could become a reusable asset for other OVS models."],"forward_implications":["For a fixed frozen backbone, the remaining headroom in training-free OVS comes mainly from building better segment-text reference sets, not from more elaborate attention or retrieval designs.","The cleaned real-image reference set is reusable: replacing FreeDA's synthetic reference data with ReME data improves retrieval-based OVS, and training-based CAT-Seg trained on ReME data beats the version trained on COCO ground-truth segment-text data.","A lightweight superpixel segmenter becomes competitive (92.3 mIoU on VOC-20 with no post-processing), meaning expensive mask generators are not needed to benefit from the refined data.","A smaller, cleaned reference set (roughly 1.0 million pairs, 4 GB) outperforms a much larger synthetic one (2.2 million pairs, 17 GB), so higher data quality also lowers storage and inference cost."],"supporting_citations":[{"why":"Supplies the class-agnostic superpixel segmenter that proposes candidate segments for both reference and test images.","marker":"[20]"},{"why":"Generates the semantically rich image descriptions whose noun phrases become candidate labels; the paper's ablations show it outperforms GT captions and BLIP-2.","marker":"[38]"},{"why":"Performs the initial segment-label pairing and provides text embeddings used throughout the pipeline.","marker":"[51]"},{"why":"Provides the visual embeddings used for intra-modal outlier filtering and for segment-segment similarity in retrieval.","marker":"[47]"},{"why":"The representative synthetic-reference retrieval baseline whose reference set and data inefficiency ReME is compared against.","marker":"[6]"},{"why":"Tip-Adapter is the source of the simple similarity-based retrieval strategy that ReME adapts.","marker":"[84]"},{"why":"COCO Stuff and COCO-2017 images are the default real-image corpus for reference set construction.","marker":"[8]"}],"fun_headline_variants":["High-quality reference data beats retrieval tricks in OVS","Data-centric design unlocks training-free segmentation","Quality over cleverness: the key to open-vocabulary segmentation","Clean segment-text pairs outperform complex retrieval","Reference set quality drives training-free OVS performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that every correctly labeled segment for a given root noun looks alike to the visual encoder, so a wrong label appears as an outlier and filtering it never removes good examples.","fun_headline_variants_meta":{"raw":{"variants":["High-quality reference data beats retrieval tricks in OVS","Data-centric design unlocks training-free segmentation","Quality over cleverness: the key to open-vocabulary segmentation","Clean segment-text pairs outperform complex retrieval","Reference set quality drives training-free OVS performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000285,"raw_usage":{"total_tokens":1665,"prompt_tokens":917,"completion_tokens":748,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":676}},"tokens_in":533,"tokens_out":748,"duration_ms":8719,"temperature":1.0,"reasoning_tokens":676,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:30:04.200915+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have humans check the segments that ReME's group-based filter drops on visually heterogeneous classes (e.g., 'furniture', 'vehicle', 'bird'): if many dropped pairs prove correctly labeled, the filter is removing good data, not misalignments. A quantitative version: replace the intra-modal filter with an oracle built from ground-truth segment-text labels; if the oracle does not reproduce most of the filtering gain (91.10 vs 70.03 on VOC-20), then the filtering mechanism claimed to carry the result is not the real driver.","supporting_citations":[{"cited_title":"Efficient graph-based image segmentation","cited_arxiv_id":null,"evidence_quote":"Supplies the class-agnostic superpixel segmenter that proposes candidate segments for both reference and test images."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Generates the semantically rich image descriptions whose noun phrases become candidate labels; the paper's ablations show it outperforms GT captions and BLIP-2."},{"cited_title":"Tip- adapter: Training-free adaption of clip for few-shot classi- fication","cited_arxiv_id":null,"evidence_quote":"Tip-Adapter is the source of the simple similarity-based retrieval strategy that ReME adapts."}],"review_version":1}