{"id":"9b0869c5-306f-4d15-80a9-372407427fba","arxiv_id":"2411.09219","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Trident, a training-free framework combining CLIP, DINO, and SAM, raises state-of-the-art open-vocabulary segmentation mIoU from 44.4 to 48.6 by splicing sub-image features and aggregating them with a SAM affinity matrix.","lead":"Researchers combined CLIP, DINO, and SAM into a training-free pipeline called Trident that stitches image-tile features before segmenting, rather than stitching the tile segmentations afterwards. On eight standard benchmarks it raises the best training-free open-vocabulary segmentation average from 44.4 to 48.6 mIoU.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The +4.2 mIoU headline depends on cross-paper baseline numbers under different per-dataset resolutions and strides; until baselines are rerun under Trident's protocol, the SOTA claim is not securely established.","rationale":"I read Trident as a training-free framework whose value depends on two things: the splice-then-segment mechanism actually increasing accuracy, and the reported numbers being a fair measure of that increase. The mechanism is supported by internal ablations (Table 3 shows Splice-then-Segment Cos beats Segment-then-Splice Cos by 6.5 mIoU on average, and Table 4 shows the affinity matrix and SAM refinement each add consistent gains), and the code is promised, so I would not attack the method purely on the positional-embedding alignment concern that the reader emphasizes; that concern is real (Eq. 3 explicitly defers the interpolation) but it is at least partially answered by these controlled ablations. The less protected step is the headline comparison itself. The per-dataset resolution settings and the decision to quote baselines 'as described in their respective publications' mean the 44.4 to 48.6 improvement is not measured under a common protocol. Table 7 demonstrates that the choice of resolution changes ProxyCLIP's score by several points, so this is not a negligible effect. Since the central claim is specifically a numerical SOTA improvement, this is the load-bearing weak point. A clear fix is to rerun all methods under Trident's stated protocol and report variance; until then, the correct verdict is conditional acceptance. I do not see a reason to reject outright, because the method is well-ablated and the paper is transparent about many implementation details, but the numerical headline needs the protocol check.","tokens_in":15666,"tokens_out":7828,"duration_ms":89174,"concrete_test":"Run the released code to re-evaluate ProxyCLIP, SCLIP, NACLIP, and Trident on all eight benchmarks under the same protocol: Trident's Sec. 4.1 resolutions and strides, same text prompts, same MMsegmentation backend, and no PAMR. If ProxyCLIP's mean under this protocol is more than about 46.0 (rather than 44.4) or if Trident's margin falls below the claimed 3.5, the headline SOTA claim is not established. Also record the epsilon value used in Eqs. 4-5 for each dataset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing condition for the central claim is not the internal splicing mechanism (which is defended by the Table 3 ablation) but the comparability of the headline numbers. Section 4.1 assigns a different input resolution and stride to each benchmark (336 for VOC20, 448 for VOC21/Object/Stuff, 576 for Context/ADE, 688 for Cityscapes), and Table 2 states that 'for other competitors, we report their performance as described in their respective publications.' The asserted 44.4 to 48.6 gain over ProxyCLIP therefore mixes Trident's tuned resolution protocol with ProxyCLIP's original protocol. This is not a cosmetic issue: Table 7 shows ProxyCLIP's own mIoU moves by up to 6.3 points with input resolution (VOC20 drops from 79.7 at 336 to 73.4 at 576), and the ProxyCLIP numbers in Table 2 (e.g., Context59 39.1, ADE 20.2) do not match Table 7's settings at any listed resolution (34.4/38.1/19.4 at 336 or 33.8/37.0/19.0 at 576). No error bars or multiple runs are provided, and the masking threshold epsilon in Eqs. 4-5 is never reported despite being a free parameter. If ProxyCLIP (or SCLIP/NACLIP) is rerun under Trident's resolutions, part of the reported margin could disappear; the paper's central 'significant improvement' claim would then need to be re-baselined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Trident, a training-free framework for open-vocabulary semantic segmentation that replaces the segment-then-splice sliding-window pipeline with a splice-then-segment paradigm. CLIP and DINO features extracted from overlapping sub-images are spliced into a global feature map, which is then aggregated through a correlation/affinity matrix derived from SAM's encoder features and attention weights, with a thresholded masking mechanism. The coarse segmentation is further refined by converting it into point, box, and mask prompts for SAM. Experiments are reported on eight benchmarks (VOC20/21, Context59/60, COCO Object/Stuff, Cityscapes, ADE20k), claiming an average mIoU of 48.6 with OpenCLIP-H/14 compared with 44.4 for the previous training-free SOTA ProxyCLIP. Component ablations in Tables 3 through 7 support the contribution of global aggregation, the affinity matrix, and SAM refinement.","tokens_in":15985,"tokens_out":6035,"duration_ms":61969,"significance":"If the reported gains survive a properly controlled comparison, Trident is a meaningful advance: it is training-free, uses no learned parameters, and shows consistent improvements across all eight benchmarks. The manuscript's main strengths are the component-level ablation tables, which support the internal design choices, and the efficiency analysis in Table 8. The central SOTA claim, however, currently rests on cross-paper baseline numbers obtained under different input resolutions and strides, so the quantitative headline is not yet secure. The proposed paradigm is clearly scoped and likely to be of interest to the dense-prediction and vision-foundation-model communities.","major_comments":[{"comment":"The headline comparison is not performed under a common evaluation protocol. Section 4.1 assigns Trident a different short-side resolution and stride for each benchmark (336 for VOC20, 448 for VOC21/Object/Stuff, 576 for Context/ADE, 688 for Cityscapes) and states that for other competitors 'we report their performance as described in their respective publications.' Since Table 7 shows ProxyCLIP's own mIoU varies with resolution (e.g., VOC20 drops from 79.7 at 336-336-112 to 73.4 at 576-336-224), and since the ProxyCLIP numbers in Table 2 (e.g., Context59 39.1, Context60 35.3, ADE 20.2) do not match any ProxyCLIP row in Table 7 (e.g., Context59 34.4-34.8, Context60 37.0-38.3, ADE 19.0-19.4), the asserted improvement from 44.4 to 48.6 may be partly a result of different resolutions and strides. Please rerun all training-free competitors under Trident's protocol, or under a single common protocol, and report those numbers in the main table.","section":"§4.1, Tables 2 and 7"},{"comment":"The load-bearing spatial alignment in the splice-then-segment step is not described. The sentence 'For brevity, the interpolation for Ifeat to align its size with A is omitted here' leaves open how sub-image features with crop-local position embeddings are placed into the global grid, how overlapping windows are combined, and what interpolation is used to match Ifeat with SAM's feature grid. This is precisely the mechanism that makes global aggregation across windows meaningful, and the paper's quantitative claims cannot be reproduced without it. Please provide the full mapping, including tensor reshaping and interpolation details, or pseudocode.","section":"§3.3, Eq. (3)"},{"comment":"The masking threshold epsilon is a free parameter that controls which token pairs contribute to the affinity matrix, yet its value is never reported in the paper or in the ablation tables. Without this value and without a sensitivity analysis, the reported gains cannot be reproduced or judged for stability. Please report epsilon for each configuration and include an ablation over epsilon.","section":"§3.3, Eqs. (4)-(5)"},{"comment":"The per-dataset resolutions and strides appear to be chosen to optimize Trident's performance (for example, stride 112 is used only for VOC20), and Table 7 shows that these choices have a large effect on results. The paper should state whether these settings were selected on the validation splits, and should report results under a fixed resolution/stride protocol as a robustness check, alongside the tuned numbers.","section":"§4.1, Table 7"}],"minor_comments":[{"comment":"There are typos: 'spatial invariance semantic' should likely be 'spatial-invariance semantics', and 'migrate' in Section 3.2 should be 'mitigate'.","section":"Abstract and §1"},{"comment":"The phrase 'morphology methods [19, 63]' is vague; please specify the connected-component labeling and morphological operations used to decompose the binary mask.","section":"§3.4"},{"comment":"The 'Avg.' column is not defined; please clarify how missing entries (e.g., SAM-CLIP has no VOC20/Context59 values) are handled when computing the average.","section":"Table 2"},{"comment":"The ProxyCLIP baseline in Table 4 (e.g., V21 59.2, ADE 19.0) differs from the ProxyCLIP row in Table 2 (V21 61.3, ADE 20.2); the text acknowledges resolution differences, but it would help to state explicitly that Table 4 uses the authors' reimplementation under Trident's protocol.","section":"Table 4"},{"comment":"Figure 1 is referenced in the introduction but is not visible in the manuscript text; please ensure the final version includes it with labeled axes.","section":"Introduction, Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is in scope for the journal and the core idea is interesting. The main revision need is the baseline protocol: the mismatched ProxyCLIP numbers between Tables 2 and 7 should be checked carefully, because if the authors' own reimplementation of ProxyCLIP under Trident's settings yields lower numbers than the published ProxyCLIP row, the headline margin will shrink. It would also be worth verifying that the hyperparameters alpha and epsilon are not selected per dataset in a way that overstates generalization."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one for the splice-then-segment reformulation. Instead of masking/segmenting each sliding-window crop independently and stitching results, the authors stitch CLIP/DINO features first and then apply a global affinity matrix derived from SAM. That is a clean, reusable pattern and I don't see it in the cited prior work. The follow-up refinement—converting CLIP masks into point/box/mask prompts for SAM—is also useful, and the ablations (Tables 3–6) support each piece consistently across eight benchmarks. Code is released. The paper earns a serious referee.\n\nThe soft spot is not the mechanism; Table 3 shows splice-then-segment beats segment-then-splice across every dataset, which addresses the worry about position-embedding misalignment when splicing crop-local features. The soft spot is evaluation comparability. Table 2 reports gains over ProxyCLIP and others using numbers \"as described in their respective publications,\" while Trident's own per-benchmark resolution and stride are tuned separately (336 for VOC20, 448 for VOC21/Object/Stuff, 576 for Context/ADE, 688 for Cityscapes). Table 7 shows ProxyCLIP is highly resolution-sensitive, moving by up to 6.3 mIoU, and the ProxyCLIP numbers in Table 2 (e.g., Context59 39.1, ADE 20.2) don't match any of Table 7's listed settings (34.4/38.1/19.4 at 336, 33.8/37.0/19.0 at 576). So the headline 44.4→48.6 mixes Trident's tuned protocol with baselines run under theirs. Also, epsilon in Eqs. 4–5 is a free parameter and never reported; the mask-prompt alpha is set from validation on two datasets; and there are no error bars or multiple runs. These are fixable, but until ProxyCLIP and friends are rerun under Trident's protocol, the \"significant improvement\" claim is not established.\n\nMy take: this is a good training-free contribution, not a breakthrough. For a CV venue, send it to review; ask for a common-protocol baseline table, reported epsilon and hyperparameter choices, and variance numbers. The OV-segmentation crowd will get real value from the splice-then-segment framing even if some of the margin shrinks.","headline":"The core splice-then-segment idea is new, decently ablated, and likely real; the +4.2 mIoU SOTA claim is not yet secure because the headline comparison mixes evaluation protocols.","tokens_in":16503,"tokens_out":2358,"would_cite":true,"duration_ms":24920,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free pipeline called Trident raises average open-vocabulary segmentation accuracy on eight benchmarks from 44.4 to 48.6 by splicing CLIP and DINO features first and then aggregating with SAM's correlations.","keywords":["open-vocabulary semantic segmentation","training-free segmentation","CLIP","DINO","SAM","splice-then-segment","affinity matrix","high-resolution segmentation"],"falsifier":"Take a high-resolution image containing repeated similar objects at different window positions. Run Trident once with the full global affinity matrix $\\mathbf{A}$ and once with $\\mathbf{A}$ replaced by a block-diagonal matrix that only mixes tokens within the same window; if the full matrix does not beat the block-diagonal version by a clear margin, the claimed cross-window aggregation is not doing the work. A direct alignment check is to register the spliced CLIP/DINO feature map against SAM's feature grid using known correspondences and measure whether the average misalignment exceeds one feature-grid cell.","tokens_in":15468,"feed_emoji":"🧩","tokens_out":12185,"duration_ms":123847,"temperature":0.7,"pith_summary":"Open-vocabulary semantic segmentation usually hits a resolution ceiling: CLIP works on small inputs, so high-resolution images are chopped into windows, segmented, and stitched back together. This paper argues that the chopping-then-stitching order is the problem, because each window sees too little context and classification fails on large objects. It proposes reversing the pipeline: extract CLIP and DINO features from the windows, splice them into one map, then aggregate that map with a global correlation matrix built from SAM's high-resolution encoder. This training-free system, called Trident, is reported to raise average mean intersection-over-union (mIoU) across eight benchmarks from 44.4 to 48.6, beating all training-free baselines and some trained ones. The claim matters because it suggests fused frozen foundation models can substantially improve dense prediction without any learned parameters.","feed_headline":"Training-free open-vocab segmentation jumps to 48.6 mIoU","feed_subtitle":"Trident stitches CLIP and DINO window features, then lets SAM's global correlations refine the map—no training needed.","key_machinery":"The load-bearing object is the affinity matrix $\\mathbf{A}$. SAM encodes the full high-resolution source image into features $\\mathbf{F}$; cosine similarity $\\mathbf{C}$ between normalized $\\mathbf{F}$ entries captures low-level visual agreement, while the attention weights $\\mathbf{W}$ from SAM's last transformer layer carry more semantic grouping but attend to background. Trident forms $\\mathbf{A}$ as $(\\mathbf{W} + \\mathbf{M}) / \\|\\mathbf{W} + \\mathbf{M}\\|$, where $M_{ij}=0$ if $C_{ij} \\ge \\epsilon$ and $M_{ij}=-W_{ij}$ otherwise, so attention to tokens with weak low-level agreement is suppressed. Multiplying the spliced CLIP/DINO feature map by $\\mathbf{A}$ performs one global aggregation whose receptive field is the entire source image rather than a single sliding window. DINO supplies object-level spatially covariant correlations inside each sub-image before splicing, and SAM's decoder refines the output using point, box, and scaled mask prompts derived from Trident's own confidence map.","core_discovery":"The paper's central claim is that the resolution failure of training-free CLIP segmentation is caused by isolated per-window processing, and that reversing the order removes the failure. In the old segment-then-splice pipeline, each sub-image's features are correlated, classified, and spliced only at the end; as source resolution rises from 336 to 688 on PASCAL VOC, ProxyCLIP's mIoU falls by up to 9.7 points because no window sees a whole object. Trident instead splices the per-window CLIP and DINO features into a single map, then applies a global affinity matrix built from SAM. The affinity matrix takes SAM's last-layer attention weights and zeros out attention to token pairs whose SAM-feature cosine similarity falls below a threshold, suppressing background links while keeping semantic ones. After one global aggregation, text embeddings classify the map, and the coarse result is converted into point, box, and mask prompts for SAM's decoder to refine. With OpenCLIP-H/14, the paper reports 48.6 average mIoU against 44.4 for the previous best training-free method.","pith_inferences":["Editorial extension: the same splice-then-segment pattern could transfer to other frozen dense-prediction models, such as open-vocabulary detection or monocular depth estimation, whenever a high-resolution encoder supplies a global correlation prior.","Editorial extension: a cheaper variant would compute the correlation matrix from a single high-resolution self-supervised encoder and drop per-window DINO guidance, isolating whether the global aggregation alone carries the gain.","Editorial extension: window size, stride, and per-dataset resolution are manual choices; a systematic sweep over them would show whether Trident's advantage persists without per-benchmark tuning."],"forward_implications":["Higher source-image resolution generally becomes an asset rather than a liability: the reported ablations show Trident's mIoU improving with resolution on most benchmarks, with VOC20 the noted exception because the underlying baseline weakens.","Window-boundary 'panel' artifacts disappear from activation and segmentation maps because the global affinity matrix lets every output token attend across the whole image.","The reported gains over ProxyCLIP are about 3.5 mIoU on average with CLIP ViT-B/16 and 4.2 with OpenCLIP ViT-H/14, split between the splice-then-segment aggregation and the SAM prompt refinement.","SAM refinement contributes roughly 1.5 mIoU on top of the aggregation and beats the PAMR post-processing baseline on Trident's cleaner masks.","All gains come without training, at the cost of lower throughput: about 10 images per second in the base configuration and 5 in the huge configuration on one RTX 4090."],"supporting_citations":[{"why":"Supplies the CLIP image and text encoders whose spatial invariance and resolution limits the paper targets; the base features in Eq. 1.","marker":"[49]"},{"why":"Supplies SAM's high-resolution encoder for the correlation matrix and the prompt-based decoder used in refinement.","marker":"[29]"},{"why":"Supplies DINO's spatially covariant features used to guide per-sub-image feature extraction.","marker":"[5]"},{"why":"ProxyCLIP is the strongest training-free baseline and contributes the cosine-similarity masking mechanism and DINO-based attention that Trident builds on.","marker":"[30]"},{"why":"MaskCLIP is the training-free baseline whose per-window feature maps form the starting point of the Splice-then-Segment ablation.","marker":"[74]"},{"why":"CLIP-DINOiser is the trained baseline that Trident compares against to show competitive performance without training.","marker":"[65]"},{"why":"NACLIP is a recent training-free neighbour method providing a comparative baseline in the same benchmark table.","marker":"[22]"}],"fun_headline_variants":["Splice-first, segment-later: training-free open-vocab hits 48.6 mIoU","No training, higher mIoU: SAM global affinity boosts open-vocab to 48.6","Trident splices CLIP and DINO, then uses SAM to lift mIoU to 48.6","Reversed pipeline: train-free open-vocab segmentation reaches 48.6 mIoU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that stitching together per-window CLIP/DINO feature maps, each computed with its own position coding, still lines up with SAM's full-image feature grid closely enough that SAM's correlation matrix can mix features across window boundaries without scrambling which pixels correspond to which locations.","fun_headline_variants_meta":{"raw":{"variants":["Splice-first, segment-later: training-free open-vocab hits 48.6 mIoU","No training, higher mIoU: SAM global affinity boosts open-vocab to 48.6","Trident splices CLIP and DINO, then uses SAM to lift mIoU to 48.6","Reversed pipeline: train-free open-vocab segmentation reaches 48.6 mIoU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000447,"raw_usage":{"total_tokens":2293,"prompt_tokens":1015,"completion_tokens":1278,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":631,"completion_tokens_details":{"reasoning_tokens":1170}},"tokens_in":631,"tokens_out":1278,"duration_ms":109445,"temperature":1.0,"reasoning_tokens":1170,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:53:34.808074+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a high-resolution image containing repeated similar objects at different window positions. Run Trident once with the full global affinity matrix $\\mathbf{A}$ and once with $\\mathbf{A}$ replaced by a block-diagonal matrix that only mixes tokens within the same window; if the full matrix does not beat the block-diagonal version by a clear margin, the claimed cross-window aggregation is not doing the work. A direct alignment check is to register the spliced CLIP/DINO feature map against SAM's feature grid using known correspondences and measure whether the average misalignment exceeds one feature-grid cell.","supporting_citations":[{"cited_title":"Segment any- thing","cited_arxiv_id":null,"evidence_quote":"Supplies SAM's high-resolution encoder for the correlation matrix and the prompt-based decoder used in refinement."},{"cited_title":"Emerg- ing properties in self-supervised vision transformers","cited_arxiv_id":null,"evidence_quote":"Supplies DINO's spatially covariant features used to guide per-sub-image feature extraction."},{"cited_title":"Proxyclip: Proxy at- tention improves clip for open-vocabulary segmentation","cited_arxiv_id":null,"evidence_quote":"ProxyCLIP is the strongest training-free baseline and contributes the cosine-similarity masking mechanism and DINO-based attention that Trident builds on."},{"cited_title":"Extract free dense labels from clip","cited_arxiv_id":null,"evidence_quote":"MaskCLIP is the training-free baseline whose per-window feature maps form the starting point of the Splice-then-Segment ablation."},{"cited_title":"Pay attention to your neighbours: Training-free open-vocabulary semantic segmentation","cited_arxiv_id":null,"evidence_quote":"NACLIP is a recent training-free neighbour method providing a comparative baseline in the same benchmark table."}],"review_version":1}