{"id":"c889c91b-05e5-4495-a32e-31abc3722d6d","arxiv_id":"2505.02075","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Interactive segmentation reveals that feature upsampler choice strongly affects dense prediction quality in frozen DINOv2, with LoftUp giving the best results.","lead":"This paper uses interactive segmentation (click-based object selection) as a new test bed for comparing feature upsamplers for vision foundation models. It finds that the LoftUp upsampler outperforms bilinear interpolation and other methods, improving segmentation quality by up to 50%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Click-injection placement was selected with a single upsampler (FeatUp's JBU) and then held fixed for all methods; an untested interaction with LoftUp's cross-attention architecture could drive the reported ranking.","rationale":"The central claim is that LoftUp and, more broadly, appropriate feature upsamplers improve frozen-VFM interactive segmentation. For this claim to be supported, the comparison must hold the evaluation protocol fixed in a way that does not favor one upsampler. Tab.2 shows the injection choice was optimized with FeatUp's JBU alone, and Sec.5.2 states that this choice was then used for all subsequent experiments. Because early versus late injection changes whether click information is present in the input to the upsampler, and because LoftUp differs architecturally from FeatUp and LiFT, the current evidence supports a conditional conclusion rather than a fully controlled ranking. The paper does have independent support: code is released, the evaluation follows standard IS protocols, and the qualitative results are consistent with the quantitative ones. I do not see an internal inconsistency or a reason to reject the work; the missing injection-type ablation is the load-bearing soft spot. The reader identified the same assumption, and the recommended verdict is unchanged: run the cross-injection ablation before accepting the ranking as definitive.","tokens_in":15438,"tokens_out":5987,"duration_ms":82054,"concrete_test":"Run the Tab.1 protocol for FeatUp and LoftUp in the two non-default injection configurations (Symmetric Patch Embedding with late injection; SimpleViT with early injection). The decisive check is whether LoftUp's NoC80 and IoU@1 advantages over FeatUp and bilinear persist on all four datasets under both alternative injections; if the ranking changes on any dataset, the headline result is confounded by the injection choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The controlled comparison in Tab.1 rests on a protocol choice made in Tab.2 and Sec.5.2: early injection for Symmetric Patch Embedding and late injection for SimpleViT were selected using FeatUp's JBU only, then applied to every upsampler. The paper does not ablate injection type for LoftUp, LiFT, or bilinear. This matters because the two injection schemes change where click information enters the pipeline: early injection modifies the low-resolution tokens consumed by the frozen VFM and therefore by the upsampler, while late injection leaves the VFM/upsampler output untouched and adds click features only at the head. LoftUp is a coordinate-based cross-attention transformer that can globally recombine the feature map, whereas FeatUp and LiFT use local kernel predictions. A method-specific sensitivity to click placement is therefore plausible. If LoftUp benefits more from early injection than FeatUp does, the headline \"up to 50% over bilinear\" could reflect an interaction between upsampler architecture and the JBU-selected injection scheme rather than general feature-upsampling quality. This is a protocol-fairness concern, not an internal inconsistency; it can be resolved by a targeted ablation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes interactive segmentation (IS) as a benchmark for evaluating feature upsampling methods on frozen vision foundation models (VFMs). It introduces a benchmark architecture that freezes the VFM and upsampler while training only a click encoder and a lightweight segmentation head, and it compares five upsampling strategies (low-resolution baseline, bilinear interpolation, LiFT, FeatUp's JBU, and LoftUp) on four IS datasets using NoC and IoU metrics. The main finding is that LoftUp yields the best performance, with up to roughly 50% improvement in NoC metrics over bilinear interpolation, and that appropriate upsamplers generally improve over the low-resolution baseline. The paper also reports unsuccessful explorations of multiscale FPN and multi-granularity configurations in the supplementary material.","tokens_in":15669,"tokens_out":8755,"duration_ms":99264,"significance":"If the main finding holds, the paper establishes interactive segmentation as a sensitive and inexpensive downstream probe for feature upsampler quality, complementing existing linear-probing benchmarks. The frozen-backbone protocol is simple and reproducible, and the code is released as iSegProbe. The performance gaps among upsamplers are large and visually interpretable. However, the central ranking currently depends on a click-injection protocol that was tuned using only FeatUp's JBU and was not independently validated for the other upsamplers; for this reason the specific significance of the LoftUp result is conditional on additional ablations.","major_comments":[{"comment":"The optimal click-injection strategy (early for Symmetric Patch Embedding, late for SimpleViT) was determined using only FeatUp's JBU and then applied to all upsamplers in Tab. 1. Because LoftUp uses a coordinate-based cross-attention transformer while FeatUp and LiFT rely on locally predicted kernels, a method-specific interaction with click placement is plausible. An ablation of early vs. late injection for at least LoftUp and LiFT is needed to rule out that the reported ranking is an artifact of a protocol tuned to FeatUp.","section":"Sec. 5.2, Tab. 2"},{"comment":"The paper reports that upsampling click features separately and merging them with upsampled image features improves FeatUp (NoC80 2.30 vs. 2.72 for late injection) but abandons this configuration for simplicity. Since this changes where click information enters the pipeline, it may interact with the upsampler architecture. The paper should either adopt this configuration uniformly across all upsamplers or provide an ablation showing that the main ranking in Tab. 1 is unchanged when separate upsampling is enabled for all methods.","section":"Sec. 5.2, Tab. 2 (separate upsampling)"},{"comment":"All quantitative results are single-run point estimates without confidence intervals or statistical tests. Even though the LoftUp improvements are large, differences between some conditions (e.g., Low-res vs. Bilinear on GrabCut) are small. For a benchmarking contribution that claims 'significant' improvements, at least three training seeds per configuration should be reported to establish that the ranking is not due to optimization noise.","section":"Sec. 5.1, Tabs. 1-5"}],"minor_comments":[{"comment":"The conclusion that the multiscale benchmark is inferior is not fully supported because the LoftUp entries are marked as 'non-final checkpoints.' Please rerun with final checkpoints or state this limitation clearly in the main text if it affects the benchmark design choice.","section":"Sec. A.2, Tab. 3"},{"comment":"Clarify how NoC is computed when a model does not reach the target IoU within the maximum of 20 clicks (e.g., is NoC set to 20, or is the instance excluded?).","section":"Sec. 5.1"},{"comment":"The hypothesis that LiFT 'shows almost no learning' would be more convincing if accompanied by training loss curves or an intermediate feature analysis, since the observed degradation relative to the low-resolution baseline is unusually severe.","section":"Sec. 5.2"},{"comment":"Minor typos: 'V olodymyr' in the author list and 'DA VIS' should be 'DAVIS'.","section":"Author list and main text"},{"comment":"The citation of reference [45] for resize-convolution may be imprecise; Odena et al. focus primarily on deconvolution artifacts, so a more direct reference for resize-convolution would help readers.","section":"Sec. 2"}],"recommendation":"major_revision","confidential_remarks":"The benchmark protocol was tuned using FeatUp's JBU only, and the winning method LoftUp shares an author with this paper. This does not by itself imply unfairness, but it raises a perception-of-fairness concern that the requested injection ablation would directly address. I would encourage the editor to ask for that ablation and also to verify that the LoftUp code and checkpoints are publicly available for reproduction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my honest take. This is a genuinely useful benchmark for feature upsamplers on frozen VFMs, and the central qualitative finding—that the choice of upsampler materially changes dense prediction quality—is likely to hold. The sharper claim, that LoftUp beats FeatUp and LiFT by a wide margin, is plausible but not fully secured because of how the click-injection protocol was chosen.\n\nWhat the paper does well: it proposes interactive segmentation as a probe for feature upsampling quality, uses a frozen backbone and frozen upsampler (a clean linear-probing style protocol), and runs the comparison over four datasets, five upsamplers, and two click encoders. Code is released. The ablation on early vs late click injection is a useful contribution by itself, and the observation that bilinear interpolation at the feature level can be worse than predicting at low resolution is a useful caution for practitioners.\n\nThe soft spot that matters: Tab. 2, where the injection-type decision is made, only tests FeatUp's JBU. That choice—early for symmetric patch embedding, late for SimpleViT—is then applied to every upsampler in Tab. 1. LoftUp's architecture is a coordinate-based cross-attention transformer, while FeatUp and LiFT rely on local kernels, so an interaction between injection placement and upsampler architecture is not a remote possibility. If LoftUp benefits more from early injection than FeatUp does, the reported 50% improvement is partly an artifact of a protocol choice. The paper should either ablate injection type for each upsampler or justify why the FeatUp-based choice is expected to transfer. This is a fixable flaw, not a deal-breaker, and it does not invalidate the benchmark architecture itself.\n\nMinor issues: no error bars (the gaps are large enough that this is not a big concern), and the failed multiscale and multi-granular explorations are described briefly, which is acceptable for a benchmark paper. Author overlap with LoftUp means the comparison should be independently checked; the code release helps.\n\nWho should read this: anyone building dense prediction heads on frozen VFMs, and researchers working on feature upsampling. It deserves a serious referee; the benchmark is a real contribution, and the confound is addressable with one additional experiment. I'd cite it for the benchmark design, not as the final answer on upsampler rankings.","headline":"Useful benchmark for feature upsamplers on VFMs, but the LoftUp advantage rests on an unverified click-injection interaction.","tokens_in":16175,"tokens_out":3081,"would_cite":true,"duration_ms":34365,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that the choice of feature upsampler strongly shapes what frozen vision foundation models can do, and that LoftUp, a coordinate-based cross-attention upsampler, beats bilinear interpolation by up to 50 percent in…","keywords":["interactive segmentation","feature upsampling","vision foundation models","LoftUp","coordinate-based cross-attention","click injection","dense prediction","benchmarking"],"falsifier":"Re-run the benchmark with every upsampler under both early and late click injection and compare the full matrix. If bilinear interpolation or FeatUp's JBU matches or beats LoftUp under its preferred injection, the headline improvement would be due to click-injection choice rather than upsampling quality.","tokens_in":15251,"feed_emoji":"🖱️","tokens_out":6436,"duration_ms":68509,"temperature":0.7,"pith_summary":"The paper sets out to show that feature upsampling is not a minor detail: for frozen vision foundation models, the method used to restore full-resolution features can change downstream dense-prediction quality substantially. It proposes interactive segmentation, where a model turns an image plus user clicks into an object mask, as a sensitive benchmark for comparing feature upsamplers. Across four datasets and two click-encoding designs, the authors find that LoftUp consistently outperforms bilinear interpolation, LiFT, and FeatUp's joint bilateral upsampler, with gains up to 50 percent reported over bilinear interpolation. If the finding holds, it implies that upsampler quality, not just backbone quality, should be treated as a first-order factor when building dense predictors from foundation models.","feed_headline":"LoftUp outdoes bilinear feature upsampling by up to 50 percent","feed_subtitle":"Click-driven segmentation reveals that a frozen vision backbone benefits sharply from how its features are upsampled.","key_machinery":"The load-bearing component is LoftUp, a coordinate-based cross-attention transformer that maps a low-resolution feature map to a full-resolution target by attending to the feature tokens from a continuous coordinate grid, trained against full-resolution pseudo-ground-truth features. The benchmark architecture around it freezes the vision backbone and the upsampler, trains only the click encoder and a three-layer segmentation head, and injects click features either early after patch embedding or late after upsampling. This setup isolates the upsampler's contribution and makes interactive segmentation a controlled probe of feature quality.","core_discovery":"The central claim is that a task-agnostic feature upsampler trained on full-resolution pseudo-ground-truth features, with global attention over a coordinate grid, can substantially improve the quality of frozen VFM features, and that interactive segmentation exposes these differences clearly. In the authors' benchmark, LoftUp outperforms the other upsamplers on every dataset and with both click encoders; for example, on GrabCut with the symmetric patch embedding encoder, the number of clicks needed to reach 80 percent IoU drops from 4.32 with bilinear feature upsampling to 1.72 with LoftUp. The paper interprets this as evidence that dense, global supervision during upsampler training matters more than local kernel prediction or fixed-factor upscaling.","pith_inferences":["The early-versus-late click-injection choice was tuned using FeatUp's JBU alone; a natural extension is to re-run the benchmark with each upsampler under both injection strategies to confirm the ranking is not an artifact of that choice.","If the result generalizes, feature upsampling quality should matter just as much for other click- or prompt-driven dense tasks, such as open-vocabulary segmentation and interactive editing.","The small gains on SBD, the training distribution, hint that the benchmark's discriminative power may come mostly from out-of-distribution datasets; testing on held-out domains deliberately would make that explicit.","One could test whether LoftUp's advantage persists with larger backbones, since the coordinate-attention budget scales with feature tokens."],"forward_implications":["Interactive segmentation can serve as a standard probing task for feature upsamplers, complementing existing dense-prediction benchmarks.","A frozen vision foundation model plus a strong upsampler plus a lightweight head can approach the performance of task-specific multiscale decoders with much lower training cost.","Upsampler design choices, especially global cross-attention and full-resolution supervision, matter more than local upsampling kernels for recovering fine detail.","Methods that only double resolution or rely on low-resolution pseudo-ground-truth targets are at a systematic disadvantage in this benchmark."],"supporting_citations":[{"why":"LoftUp, the central upsampler that achieves the best results in the benchmark.","marker":"[26]"},{"why":"FeatUp, a baseline upsampler and the source of the JBU variant used to select click-injection strategies.","marker":"[18]"},{"why":"LiFT, the fixed-factor local-kernel upsampler that performs worst among learnable methods.","marker":"[57]"},{"why":"SimpleClick, source of the symmetric patch embedding click encoder, training protocol, and evaluation setup.","marker":"[37]"},{"why":"DINOv2, the frozen vision backbone whose features are upsampled in all main experiments.","marker":"[46]"},{"why":"SimpleViT, the alternative click encoder used with late injection.","marker":"[6]"},{"why":"RITM, source of the training-time click simulation and evaluation protocol.","marker":"[56]"},{"why":"SBD, the dataset used for training and one of the four evaluation datasets.","marker":"[22]"},{"why":"GrabCut, the primary evaluation dataset where the largest LoftUp gains are observed.","marker":"[54]"}],"fun_headline_variants":["LoftUp upsamples better: slashes clicks to 80% IoU","Interactive segmentation shows LoftUp tops VFM upsampling","LoftUp: fewer clicks for 80% IoU on GrabCut","Feature upsampling matters: LoftUp boosts frozen VFM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the click-injection strategy chosen once with a single upsampler, early for one click encoder and late for the other, is also the best strategy for the other upsamplers it compares, so its ranking is not an artifact of mismatched click placement.","fun_headline_variants_meta":{"raw":{"variants":["LoftUp upsamples better: slashes clicks to 80% IoU","Interactive segmentation shows LoftUp tops VFM upsampling","LoftUp: fewer clicks for 80% IoU on GrabCut","Feature upsampling matters: LoftUp boosts frozen VFM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000374,"raw_usage":{"total_tokens":1960,"prompt_tokens":873,"completion_tokens":1087,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":1008}},"tokens_in":489,"tokens_out":1087,"duration_ms":9216,"temperature":1.0,"reasoning_tokens":1008,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T01:01:48.191126+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the benchmark with every upsampler under both early and late click injection and compare the full matrix. If bilinear interpolation or FeatUp's JBU matches or beats LoftUp under its preferred injection, the headline improvement would be due to click-injection choice rather than upsampling quality.","supporting_citations":[{"cited_title":"Loftup: Learning a coordinate-based feature upsampler for vision foundation models, 2025","cited_arxiv_id":null,"evidence_quote":"LoftUp, the central upsampler that achieves the best results in the benchmark."},{"cited_title":"Brandt, Axel Feld- mann, Zhoutong Zhang, and William T","cited_arxiv_id":null,"evidence_quote":"FeatUp, a baseline upsampler and the source of the JBU variant used to select click-injection strategies."},{"cited_title":"Lift: A surprisingly simple lightweight feature transform for dense vit descriptors","cited_arxiv_id":null,"evidence_quote":"LiFT, the fixed-factor local-kernel upsampler that performs worst among learnable methods."},{"cited_title":"Simpleclick: Interactive image segmentation with simple vi- sion transformers","cited_arxiv_id":null,"evidence_quote":"SimpleClick, source of the symmetric patch embedding click encoder, training protocol, and evaluation setup."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DINOv2, the frozen vision backbone whose features are upsampled in all main experiments."},{"cited_title":"Petrov, and Anton Konushin","cited_arxiv_id":null,"evidence_quote":"RITM, source of the training-time click simulation and evaluation protocol."},{"cited_title":"Bourdev, Subhransu Maji, and Jitendra Malik","cited_arxiv_id":null,"evidence_quote":"SBD, the dataset used for training and one of the four evaluation datasets."},{"cited_title":"”grabcut”: interactive foreground extraction using iterated graph cuts","cited_arxiv_id":null,"evidence_quote":"GrabCut, the primary evaluation dataset where the largest LoftUp gains are observed."}],"review_version":1}