{"id":"d9d88236-ce34-4c11-b558-0edb49f25aae","arxiv_id":"2507.11030","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The paper defines the personalized open-vocabulary segmentation task and shows a plug-in method with negative mask proposal that improves personal-concept IoU while roughly preserving original open-vocabulary performance.","lead":"This paper introduces personalized open-vocabulary semantic segmentation, a task where a model learns to segment a user's own object (such as 'my mug cup') from a few image and mask examples while still handling arbitrary text classes. The authors propose a plug-in method with a learnable text prompt, a negative mask proposal, and visual embedding injection, and they evaluate it on three new benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'maintaining OVSS performance' half of the central claim is not established: mIoU on FSSper/CUBper is scored against the evaluated backbone's own predictions, and Table 1 shows consistent ODISE mIoU drops where real labels exist.","rationale":"The paper's technical contribution is a plug-in that improves IoUper on a new task, and the ablation study plausibly attributes the gain to the negative mask and visual injection. I do not see an internal flaw that would overturn the method itself. The load-bearing weakness is the preservation clause: on FSSper and CUBper the mIoU metric is computed against pseudo-labels produced by the same backbone whose original performance is being claimed as preserved, so it cannot establish open-vocabulary accuracy; on ADEper the real-label numbers show small but consistent ODISE mIoU drops, and no error bars are provided. This matches the reader's weakest_assumption, with the added observation that the own Table 1 numbers for ODISE already strain the 'without compromising' wording. I also noted a secondary inconsistency: Table 4 is captioned CUBper with K=1 but the Ours row (76.80 IoUper, 84.51 precision, 89.07 recall) is exactly the K=5 ablation row from Table 2, which would be an unfair comparison against K=1 baselines. These issues are addressable with human-annotated or independent pseudo-labels, error bars, and a corrected Table 4, so the CONDITIONAL verdict stands unchanged.","tokens_in":15752,"tokens_out":10516,"duration_ms":124038,"concrete_test":"Annotate open-vocabulary class labels on a held-out subset of FSSper and CUBper (e.g., 50 classes × 5 positive/5 negative images) and recompute mIoU with these human labels for SAN and ODISE with and without the plug-in, reporting per-seed variance. If the recomputed mIoU shows a statistically significant drop for either backbone, or if the human-label mIoU differs materially from the pseudo-label mIoU, the 'maintaining OVSS performance' claim fails. A cheaper cross-check is to generate pseudo-labels with an independent OVSS model (e.g., MaskCLIP) and see whether the FSSper/CUBper mIoU conclusions change.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two parts: improve IoUper and preserve OVSS mIoU. The preservation part is not adequately measured. On FSSper and CUBper, no open-vocabulary ground truth exists, so the paper uses the predictions of the pretrained OVSS backbone itself as pseudo-labels for mIoU (Sec. 4.1, Evaluation Metrics; Supp. A.2). Because the baseline in Table 1 is that same backbone, the pseudo-label mIoU mainly measures agreement with the unpersonalized model, not open-vocabulary segmentation accuracy; a method that leaves all non-personal predictions unchanged would score near-perfectly on those regions regardless of true semantics. Thus the 'does not compromise the original OVSS performance' claim is unsupported on two of the three benchmarks. On ADEper, where real labels are available, Table 1 shows ODISE mIoU dropping from 23.86 to 22.48 (FSSper), 47.48 to 44.99 (CUBper), and 12.22 to 12.19 (ADEper) after adding our method; without error bars these drops are at least inconsistent with the abstract's unconditional 'without compromising' claim. The IoUper improvements and the SAN results are credible; the preservation claim is the soft spot.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new task, personalized open-vocabulary semantic segmentation (OVSS), in which a model must segment a user-provided personal concept (e.g., \"my mug cup\") in addition to performing standard open-vocabulary segmentation. To address this task, the authors propose a plug-in method built on text prompt tuning, augmented with a negative mask proposal that captures non-personal concepts and with visual embeddings injected into the textual prompt. The method is applied to two existing OVSS models (SAN and ODISE) and evaluated on three newly constructed benchmarks: FSS^per, CUB^per, and ADE^per, built from FSS-1000, CUB-200, and ADE-20K respectively. The paper reports consistent improvements in IoU^per across backbones and datasets, and claims that the original OVSS mIoU is maintained.","tokens_in":16001,"tokens_out":6232,"duration_ms":66378,"significance":"The task is well motivated and timely: existing OVSS models cannot distinguish a user-specific instance from other objects of the same category, and few-shot segmentation methods lack open-vocabulary flexibility. The proposed method is simple, parameter-efficient (about 0.4M trainable parameters), and architecture-agnostic, and the ablation study in Table 2 clearly isolates the contribution of each component. The paper also provides a useful set of benchmarks for a new task. However, the evaluation of the \"maintaining OVSS performance\" half of the central claim is weakened by the use of pseudo-labels from the pretrained model itself on two of the three benchmarks, and the comparison with few-shot methods in Table 4 contains a mislabeled experimental setting. These issues are fixable and do not undermine the core method's potential, but they do affect the strength of the stated claims.","major_comments":[{"comment":"The mIoU metric on FSS^per and CUB^per is computed against the pretrained OVSS model's own predictions as pseudo ground truth (stated in Sec. 4.1 and again in Supp. A.2). Since the baseline in Table 1 is the same pretrained model, these mIoU scores primarily measure agreement with the unpersonalized model, not open-vocabulary segmentation accuracy. A method that leaves all non-personal predictions unchanged would score highly by construction, so the claim in Sec. 4.2 that \"our method maintains mIoU\" is not established for two of the three benchmarks. On ADE^per, which has real labels, Table 1 shows a small drop for ODISE (12.22 to 12.19); without error bars this does not support an unconditional \"without compromising\" statement. Please either evaluate mIoU on a benchmark with human labels, or explicitly restrict the preservation claim to datasets with real labels and report variance.","section":"Sec. 4.1 (Evaluation Metrics); Supp. A.2"},{"comment":"The comparison with few-shot methods is mislabeled. Table 4's caption says \"CUBper (K = 1)\" but the reported \"Ours\" value (76.80) matches the K=5 row in Table 1; the corresponding K=1 value is 76.70. This makes the claimed advantage over SEEM and SegGPT at K=1 inaccurate. Please correct the table or clarify the experimental setup, and if the comparison is at K=5, also report the baseline methods' performance at K=5.","section":"Table 4"},{"comment":"The quantitative results lack error bars or multiple runs, and hyperparameters (alpha, lambda_neg_M, lambda_neg_Z, learning rates) are tuned separately per dataset and per backbone, as listed in Supp. A.1. For small differences such as the ODISE ADE^per mIoU change (12.22 to 12.19), it is impossible to tell whether the method preserves performance or slightly degrades it. Please report standard deviations over multiple random support-set selections and state whether the chosen hyperparameters are constant across classes or selected per dataset; a sensitivity analysis would help support the claim of consistent improvement.","section":"Table 1 and Supp. A.1"}],"minor_comments":[{"comment":"The title contains an erroneous space in \"Open-V ocabulary\"; it should be \"Open-Vocabulary\".","section":"Title"},{"comment":"The caption reads \"SAN wihtout personalization\"; \"wihtout\" should be \"without\".","section":"Supp. B.2, Fig. 11 caption"},{"comment":"The text says \"we select α = 0.1 since it achives the best IoUper\"; \"achives\" should be \"achieves\".","section":"Supp. B.1"},{"comment":"The caption uses \"Quantative results\"; it should be \"Quantitative results\".","section":"Supp. B.4, Table 7 caption"},{"comment":"The normalization in Eq. (6) is hard to read: the denominator is written with a sum over 1(M'_gt = 1) but with a leading \"1\" that appears to be an indicator; please clarify the notation explicitly.","section":"Eq. (6)"},{"comment":"In the sentence \"by average, we improve IoUper of SAN by ...\", the phrase \"by average\" should be \"on average\".","section":"Sec. 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant new task and the proposed method is reasonable, but the preservation claim is not rigorously supported due to the pseudo-label mIoU evaluation on two of the three benchmarks. I would recommend requiring a revision that either adds a real-label mIoU evaluation or substantially tempers the claim, and that corrects the Table 4 inconsistency. The authors should also consider releasing the benchmark construction code and data splits to support reproducibility, as the paper currently does not mention code or data release."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, the task it defines—personalized open-vocabulary semantic segmentation—is new and useful: given a few image-mask pairs of 'my mug cup,' the model should find that object among same-class distractors while still doing ordinary open-vocabulary segmentation. Second, its method is simple and it works: text prompt tuning plus a 'negative mask proposal' that learns to fire on everything else, plus a visual-embedding injection. On FSSper, CUBper, ADEper it lifts IoUper substantially for both SAN and ODISE, even at K=1, and the ablation in Table 2 supports the role of each piece.\n\nThe smart part is the negative mask. Plain prompt tuning increases false positives—the model calls every bird 'my bird.' The negative mask learns a complementary mask supervised by 1 - Mgt and an embedding supervised to match the uniform distribution over the other open-vocabulary classes. That is a clean, cheap fix, and the precision/recall breakdown in Table 2 shows it does what it claims.\n\nWhere it goes soft is the second half of the central claim: that the method maintains original OVSS performance. For FSSper and CUBper there are no open-vocabulary ground-truth labels, so the paper computes mIoU against the pretrained OVSS model's own predictions as pseudo-labels. The authors say this explicitly in Sec. 4.1. That makes 'maintaining performance' on those two datasets mostly a measure of agreement with the unpersonalized model, not true segmentation accuracy. On ADEper, where real labels exist, the ODISE mIoU drop is small (12.22 to 12.19) but not zero, and on the two pseudo-labeled datasets the ODISE drops are larger (23.86 to 22.48, 47.48 to 44.99). The abstract's unqualified 'without compromising' is therefore stronger than the evidence. For SAN the mIoU is essentially flat or slightly better, so the method seems lighter on that backbone, but the claim should be scoped per backbone.\n\nOther soft spots are evaluative rather than fatal: no error bars or repeated runs, hyperparameters like alpha and lambda_neg_M are tuned per dataset (values differ by up to 500x), and no code or benchmark release. A held-out validation split or at least a fuller sensitivity discussion would make the per-dataset constants less alarming. The comparisons to SEEM, SegGPT, and kNN-CLIP are useful but only on CUBper.\n\nWho gets value: anyone working on personalization, few-shot segmentation, or assistive robotics. The task formulation alone is worth discussing. The IoUper gains are credible and the failure mode it targets is real. But the evaluation needs tightening—especially the mIoU measurement and per-dataset tuning—before the 'without compromising' line can stand. I'd send it to peer review with a request for major revisions, not desk-reject it.","headline":"A genuinely new task and a simple plug-in that works, but the 'without compromising' claim is only half-measured; worth a serious referee with revisions required.","tokens_in":16600,"tokens_out":3055,"would_cite":true,"duration_ms":32613,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adding a negative mask and visual-embedding injection to text prompt tuning lets open-vocabulary segmentation models learn a user's personal object from a few image-mask pairs without losing original accuracy.","keywords":["personalized open-vocabulary semantic segmentation","text prompt tuning","negative mask proposal","visual embedding injection","few-shot segmentation","CLIP","open-vocabulary segmentation","instance-level personalization"],"falsifier":"Recompute mIoU on FSS$^\\text{per}$ and CUB$^\\text{per}$ using human-annotated ground-truth labels for every open-vocabulary class instead of treating SAN or ODISE predictions as ground truth; if the gap between personalized and unpersonalized mIoU widens beyond the paper's reported near-flat differences, the 'maintains original performance' claim fails. A second check: run the method on pairs of near-identical instances of the same class and count how often the wrong instance is labelled as the personal concept.","tokens_in":15530,"feed_emoji":"🎯","tokens_out":7328,"duration_ms":71547,"temperature":0.7,"pith_summary":"Open-vocabulary semantic segmentation (OVSS) models can label arbitrary classes described in text, but they cannot learn which particular object is 'mine'—the mug, bird, or dog that belongs to a specific user. This paper defines the task of personalized OVSS: given a few pairs of images and masks of one personal concept, the model must segment that concept when it appears among other objects of the same class, while keeping its original open-vocabulary accuracy. The proposed plug-in method combines text prompt tuning with a negative mask proposal and visual-embedding injection, and it is evaluated on three new benchmarks built from FSS-1000, CUB-200, and ADE-20K. The paper reports higher personalized IoU on both SAN and ODISE backbones with roughly unchanged mIoU, which is what the task's dual goal requires.","feed_headline":"Plug-in finds 'my mug' among many without forgetting other labels","feed_subtitle":"A few image-mask pairs teach an open-vocabulary model your specific object while preserving accuracy.","key_machinery":"The load-bearing mechanism is the negative mask proposal. The paper adds a learnable linear layer $W_Z$ that combines the existing mask embeddings into one negative embedding $Z_{\\text{neg}}$ and a learnable convolution $W_M$ that combines the existing mask proposals into one negative mask $M_{\\text{neg}}$. The negative embedding is trained with a loss that spreads probability over all classes except the personal concept, and the negative mask is trained against the complement $1 - M_{\\text{gt}}$ of the personal ground-truth mask. A second mechanism, visual embedding injection, replaces the learnable textual prompt $T_{\\text{per}}$ with the interpolation $\\alpha \\cdot F_{\\text{per}} + (1-\\alpha) \\cdot T_{\\text{per}}$, where $F_{\\text{per}}$ is the CLIP feature of the personal-concept pixels averaged under the ground-truth mask. Together they enforce the paper's core observation: reducing false positives is what makes prompt tuning safe for personal concepts.","core_discovery":"The central claim is that a personal visual concept can be grafted onto a pretrained OVSS model by tuning a single learnable textual embedding for the concept, adding a negative mask proposal that learns everything except the concept, and interpolating that textual embedding with a mask-averaged visual embedding of the concept. The negative mask proposal is the key correction: it consists of one extra mask embedding and one extra mask, supervised respectively to spread probability uniformly over all non-personal classes and to predict the complement of the personal ground-truth mask. This counteracts the false positives that plain text prompt tuning produces when another instance of the same class appears. Across the new FSS$^\\text{per}$, CUB$^\\text{per}$, and ADE$^\\text{per}$ benchmarks, the method raises SAN's average IoU$^\\text{per}$ from 41.08 to 53.56 on FSS$^\\text{per}$ and from 6.88 to 22.67 on ADE$^\\text{per}$, with mIoU nearly flat, and it also improves ODISE. The paper frames this as the first task definition, method, and benchmark suite for personalized OVSS.","pith_inferences":["The paper's mIoU figures on FSS$^\\text{per}$ and CUB$^\\text{per}$ treat the pretrained OVSS model's own predictions as ground truth, so a stricter human-annotated evaluation could either confirm or overturn the claim that original performance is maintained.","The negative-mask idea is a general cure for the overconfidence that comes with learning an individual instance; it could transfer to referring expression segmentation or personalized visual question answering wherever same-class distractors dominate the errors.","The benchmarks mostly contain visually distinguishable classes, so a natural stress test is near-duplicate instances of the same object, such as two identical mugs; the paper's concat-dataset experiment hints at this but does not cover true near-duplicates.","The method requires masks at personalization time; replacing them with clicks or boxes from an interactive segmenter and measuring the IoU$^\\text{per}$ drop would show how far the plug-in can go with weaker supervision."],"forward_implications":["The plug-in works on two structurally different OVSS backbones, SAN and ODISE, and improves personalized IoU even with a single image-mask pair, so the mechanism does not depend on a particular architecture.","Because the negative mask is trained to fire on everything outside the personal concept, the method avoids the labor-intensive collection of hard-negative images of the same class that earlier personalization approaches required.","With only about 0.4M trainable parameters and 200 tuning iterations, personalization is cheap enough to be run per user or per device rather than as a global retraining step.","A personalized prompt can be composed with ordinary text prompts, so the same learned concept can be used to describe things like 'hat on my dog', extending reference from the object to its parts."],"supporting_citations":[{"why":"Supplies the CLIP text and image encoders used to initialize the learnable prompt and to compute the injected visual embedding.","marker":"[32]"},{"why":"The SAN backbone the plug-in is applied to and evaluated on across all three benchmarks.","marker":"[40]"},{"why":"The ODISE backbone the plug-in is applied to and evaluated on across all three benchmarks.","marker":"[38]"},{"why":"Source of the FSS-1000 classes and foreground masks that form the FSS$^\\text{per}$ benchmark.","marker":"[21]"},{"why":"Source of the CUB-200 bird classes and masks that form the CUB$^\\text{per}$ benchmark.","marker":"[34]"},{"why":"Source of the ADE-20K scenes and labels that form the ADE$^\\text{per}$ benchmark.","marker":"[44]"},{"why":"Earlier evidence that positive-only personalization of a multimodal assistant over-predicts the target subject, motivating the negative mask proposal.","marker":"[29]"},{"why":"A few-shot segmentation baseline (SegGPT) that the paper shows suffers low precision on same-class distractors.","marker":"[36]"},{"why":"A few-shot segmentation baseline (SEEM) that the paper shows suffers low precision on same-class distractors.","marker":"[49]"}],"fun_headline_variants":["Plug-in adds 'my' concepts to OVSS with few mask pairs","New plug-in personalizes open-vocabulary segmentation","Negative mask proposal cuts false positives in personal OVSS","Graft personal objects onto OVSS via prompt tuning","Few image-mask pairs teach OVSS your specific object"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation of 'maintaining original OVSS performance' on FSS$^\\text{per}$ and CUB$^\\text{per}$ assumes that the open-vocabulary model's own predictions are the correct labels for all non-personal classes, so if those predictions are biased or wrong, the mIoU claim is not actually measuring the model's true accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Plug-in adds 'my' concepts to OVSS with few mask pairs","New plug-in personalizes open-vocabulary segmentation","Negative mask proposal cuts false positives in personal OVSS","Graft personal objects onto OVSS via prompt tuning","Few image-mask pairs teach OVSS your specific object"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1537,"prompt_tokens":1013,"completion_tokens":524,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":439}},"tokens_in":629,"tokens_out":524,"duration_ms":6201,"temperature":1.0,"reasoning_tokens":439,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:18:03.730444+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute mIoU on FSS$^\\text{per}$ and CUB$^\\text{per}$ using human-annotated ground-truth labels for every open-vocabulary class instead of treating SAN or ODISE predictions as ground truth; if the gap between personalized and unpersonalized mIoU widens beyond the paper's reported near-flat differences, the 'maintains original performance' claim fails. A second check: run the method on pairs of near-identical instances of the same class and count how often the wrong instance is labelled as the personal concept.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP text and image encoders used to initialize the learnable prompt and to compute the injected visual embedding."},{"cited_title":"Side adapter network for open-vocabulary semantic segmentation","cited_arxiv_id":null,"evidence_quote":"The SAN backbone the plug-in is applied to and evaluated on across all three benchmarks."},{"cited_title":"Open-vocabulary panop- tic segmentation with text-to-image diffusion models","cited_arxiv_id":null,"evidence_quote":"The ODISE backbone the plug-in is applied to and evaluated on across all three benchmarks."},{"cited_title":"Fss-1000: A 1000-class dataset for few- shot segmentation","cited_arxiv_id":null,"evidence_quote":"Source of the FSS-1000 classes and foreground masks that form the FSS$^\\text{per}$ benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the CUB-200 bird classes and masks that form the CUB$^\\text{per}$ benchmark."},{"cited_title":"Scene parsing through ade20k dataset","cited_arxiv_id":null,"evidence_quote":"Source of the ADE-20K scenes and labels that form the ADE$^\\text{per}$ benchmark."},{"cited_title":"Yo’llava: Your personalized lan- guage and vision assistant, 2024","cited_arxiv_id":null,"evidence_quote":"Earlier evidence that positive-only personalization of a multimodal assistant over-predicts the target subject, motivating the negative mask proposal."},{"cited_title":"Seggpt: Segmenting ev- erything in context","cited_arxiv_id":null,"evidence_quote":"A few-shot segmentation baseline (SegGPT) that the paper shows suffers low precision on same-class distractors."},{"cited_title":"Segment everything everywhere all at once","cited_arxiv_id":null,"evidence_quote":"A few-shot segmentation baseline (SEEM) that the paper shows suffers low precision on same-class distractors."}],"review_version":1}