{"id":"976b30fb-8bdf-4427-959f-b410aadacbc5","arxiv_id":"2412.18806","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"FOR fine-tunes a CLIP encoder with a learnable-query decoder head and a pseudo-label loss, improving open-vocabulary object-level retrieval by up to 8 mAP@50 points over prior art.","lead":"A team from General Motors presents FOR, a method that fine-tunes CLIP for open-vocabulary image retrieval where the query is an object name. The key idea is a new decoder-style head with a small set of learnable \"queries\" that summarize the objects in an image, trained with a combination of ordinary labels and pseudo-labels to preserve CLIP's ability to handle new categories.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pseudo-label threshold is tuned on the evaluation protocol and noise is never quantified, so the reported novel-category gains may be inflated.","rationale":"The reader's weakest assumption concerns pseudo-label accuracy. I agree this is the critical link. The pseudo-label loss is not a minor regularizer: without it, supervised-only FOR catastrophically forgets novel classes (COCO novel 21.51 vs 89.17 in Table 1). Therefore, the quality and calibration of pseudo-labels directly determines the main claim. The paper's own Supplement D admits 'erroneous labels,' including associative errors (airplane -> 'Parisian') and context errors, but no quantitative precision is reported. Furthermore, the threshold 5e-4 is set by hyper-parameter search, with no information about which split was used. If the val split was used for this search, the model's training signal is tuned to the exact queries used for evaluation, inflating the reported mAP. Even if the search was on the train split, the paper does not show sensitivity to this hyperparameter; a method whose gains depend on a precisely tuned noise floor is brittle. The proposed test—selecting the threshold on a disjoint split or fixing it across datasets—directly addresses this. Should the gains persist under this test, the concern is resolved; if not, the verdict should be at most CONDITIONAL, pending a more robust evaluation protocol.","tokens_in":125,"tokens_out":14317,"duration_ms":142628,"concrete_test":"Retrain FOR on COCO with the pseudo-label threshold selected on a held-out 10% split of the training set (never the val set), and repeat with a single fixed threshold of 5e-4 across all three datasets. Compare novel-category mAP@50 against Cluster-CLIP. If the gain on any dataset drops by more than 2 points relative to the reported numbers, the results are partly an artifact of evaluation-set tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central mechanism is the pseudo-label loss (Section 3.2, Eq. 4): it is the sole driver of novel-category gains, since supervised-only training collapses novel mAP (Table 1: COCO novel 21.51 vs 89.17 with both losses). The authors admit in Supplement D that pseudo-labels are erroneous, citing associative errors (airplane images yielding 'Parisian' labels) and context errors, yet they never quantify the noise. Moreover, the confidence threshold (5e-4) is chosen by 'hyper-parameter search' without stating the split; if the COCO/LVIS/nuImages val sets were used to select it, the noise level is optimized for the exact evaluation categories. With 21K classes, a threshold of 5e-4 can generate many false positives, and the paper reports no sensitivity analysis. If the gains vanish when the threshold is selected on a disjoint split or fixed across datasets, the central claim of up to 8 mAP@50 over SoTA is not supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FOR, a fine-tuning framework for object-centric open-vocabulary image retrieval (OC-OVIR). FOR modifies CLIP's last attention layer into a SUM-CLIP head with learnable queries and decoder layers, producing a small number of image embeddings without CPU clustering. Training combines a supervised loss on base categories with a pseudo-label loss generated by Cluster-CLIP over ImageNet-21K classes. Experiments on COCO, LVIS, and nuImages report consistent improvements over Cluster-CLIP, especially on novel categories, with 50 embeddings per image and no CPU post-processing. The paper also reports semi-supervised results showing that pseudo-labels alone can maintain most of the novel-category performance under very low labeling rates.","tokens_in":21223,"tokens_out":4324,"duration_ms":38047,"significance":"If the reported results hold, FOR is a practically relevant advance: it shows that fine-tuning for OC-OVIR can improve novel-category retrieval while retaining a compact, dual-encoder representation, and it provides a concrete recipe (SUMs-CLIP head plus two-loss training) that outperforms the existing non-finetuned state of the art. The paper's strengths include systematic comparisons across three datasets, ablations of the head design, freezing strategy, query count, decoder depth, and pseudo-label source, and an explicit semi-supervised evaluation. The central weakness is that the pseudo-label mechanism, which is the sole driver of the novel-category gains, depends on a confidence threshold whose selection protocol is under-specified and whose noise is acknowledged but never quantified. This leaves the headline improvement over Cluster-CLIP not fully established.","major_comments":[{"comment":"The pseudo-label confidence threshold is load-bearing: Table 1 shows that removing the pseudo-label loss collapses COCO novel mAP@50 from 89.17 to 21.51, and Section 4.2 states only that the threshold was \"chosen through an hyper-parameter search to be 5e−4\" without specifying the split used. If the COCO, LVIS, or nuImages validation sets were used for this search, the reported novel-category gains are tuned to the evaluation protocol. Please specify the validation procedure and provide a sensitivity analysis over the threshold (e.g., 1e−5 to 1e−2) either on a disjoint split or with a single threshold fixed across all three datasets, reporting the resulting mAP@50 for base and novel categories.","section":"Section 4.2 and Section 3.2 (Eq. 4)"},{"comment":"The pseudo-labels are generated by Cluster-CLIP, which is also the primary baseline in Tables 1 and 3. The paper acknowledges in Supplement D that pseudo-labels contain errors, including association errors (an airplane image yielding 'AirFrance' then 'Parisian') and context errors, but it never quantifies the noise rate. Because the pseudo-label loss is the mechanism responsible for the novel-category gains, the reported improvements could in part reflect systematic biases in the teacher rather than genuine open-vocabulary ability. Please report a quantitative evaluation of pseudo-label quality (e.g., precision/recall against a held-out sample of ground-truth annotations, or the fraction of images receiving at least one erroneous label) and, if possible, an experiment with corrupted or randomly replaced pseudo-labels to show how performance degrades.","section":"Section 3.2 and Supplementary Section D"},{"comment":"The claim of \"up to 8 mAP@50 points\" over state of the art is based on comparisons where FOR is fine-tuned on the target dataset while Cluster-CLIP is not. This is a legitimate comparison for the OC-OVIR setting, but the paper should state explicitly that the improvement is achieved under the assumption that target-dataset base labels are available for fine-tuning. For practitioners, this is an important caveat, and the abstract and introduction should be clearer that the gains apply to the fine-tuning setting, not to zero-shot retrieval from a frozen model.","section":"Table 1 and Section 4.3"}],"minor_comments":[{"comment":"The abstract says \"up to 8 mAP@50 points,\" while Section 4.3 says \"up to 7.6 mAP@50 points on novel categories and 10.3 on base categories.\" These numbers should be harmonized.","section":"Abstract and Section 4.3"},{"comment":"The notation \"NX j=1\" is a typographical artifact and should be rendered as a proper summation subscript.","section":"Equation (5)"},{"comment":"The reference for Cluster-CLIP lists the venue as \"34th British Machine Vision Conference 2022\" but the publication year is given as 2023; please correct the year or venue label.","section":"Reference [38]"},{"comment":"The sentence \"green, orange, and red, indicating labels that exists, might exists, and do not exists in the image\" contains grammatical errors and should be rewritten for clarity.","section":"Supplementary Section D"},{"comment":"The row labels \"none,\" \"v,o,\" and \"q,k,v,o\" are ambiguous because the column header is \"frozen\"; it would be clearer to label the rows explicitly as \"layers frozen: none,\" \"layers frozen: v,o,\" and \"layers frozen: q,k,v,o.\"","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a practical task and the proposed method is coherent, but the evaluation protocol for the key hyperparameter (pseudo-label threshold) is under-specified. If the authors can provide a disjoint validation split and a threshold sensitivity analysis, and quantify pseudo-label noise, the central claim would be convincing. I do not see a fundamental flaw in the architecture or training objective; the issues are load-bearing but fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:2412.18806.\n\nThe paper does something genuinely useful: it turns object-centric open-vocabulary retrieval into a fine-tunable problem. The SUM-CLIP head—learnable queries plus a few decoder layers before CLIP's final attention—gives a compact per-image representation without the CPU clustering Cluster-CLIP depends on, and the two-loss schedule (supervised on base categories plus pseudo-labels from ImageNet-21K via Cluster-CLIP) is a reasonable way to fight open-vocabulary forgetting. The gains are consistent across COCO, LVIS, and nuImages, and the ablations are honest: supervised-only training collapses novel categories on COCO and nuImages, pseudo-labels alone recover much of the gap, and freezing the attention linear layers matters. The supplement's comparison with detection frameworks is a nice practical addition.\n\nThe soft spots are real, though not fatal. The pseudo-label confidence threshold (5e-4) is chosen by hyper-parameter search without saying which split was used. If the val sets guided the choice, the reported novel-category differences are partially tuned to the test protocol. The pseudo-labels themselves are admittedly noisy (the supplement shows airplane→'Parisian' type errors), and the noise is never quantified. There is also a circularity concern, since Cluster-CLIP supplies the pseudo-labels and is the baseline being compared against. That is mitigated by the supervised branch providing independent signal, but it means the novel-category numbers should be read as \"distilled from Cluster-CLIP plus base supervision\" rather than as pure evidence that the model sees novel objects better than CLIP.\n\nThe paper also lacks code and error bars, so the precision of the headline numbers is unverifiable. For a paper this empirical, I'd want at least the threshold sensitivity analysis and a fixed threshold across datasets before trusting the 8-point claim.\n\nStill, this deserves peer review. The method is well-motivated, the experiments are systematic, and the authors are transparent about limitations. A reviewer can reasonably ask for code, a threshold-sweep on held-out data, and a noisy-pseudo-label analysis; the core idea is strong enough that those requests are addressable. If I were the editor, I'd send it out.","headline":"Competent, well-ablated fine-tuning method for object-centric open-vocabulary retrieval with genuine gains, but the headline novel-class improvements rest on a pseudo-label threshold tuned without a clear split and noisy pseudo-labels that are never quantified.","tokens_in":21805,"tokens_out":2519,"would_cite":true,"duration_ms":22529,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FOR fine-tunes CLIP's image encoder with a learnable-query head and a pseudo-label loss so that object-centric open-vocabulary retrieval improves by up to 8 mAP@50 points over the previous state of the art, using only 50 embeddings per…","keywords":["open-vocabulary image retrieval","object-centric retrieval","CLIP fine-tuning","learnable queries","pseudo-labels","semi-supervised learning","ImageNet-21K","image retrieval"],"falsifier":"Annotate a random sample of the pseudo-labels accepted at the $5 \\times 10^{-4}$ threshold and measure their precision; then retrain FOR with those labels removed or corrected. If the novel-category mAP@50 gain over Cluster-CLIP survives the correction, or vanishes when the labels are removed, the paper's proposed mechanism—that pseudo-label knowledge transfer drives the gain—would be supported or refuted.","tokens_in":1894,"feed_emoji":"🔍","tokens_out":5013,"duration_ms":109185,"temperature":0.7,"pith_summary":"FOR is a fine-tuning recipe for object-centric open-vocabulary image retrieval: instead of using a frozen CLIP model with post-processing, it adapts CLIP's image encoder to a target dataset while trying to preserve the vision-language alignment that makes open-vocabulary queries work. The paper claims that its SUM-CLIP head, which produces a fixed small set of representative image embeddings through learnable queries and decoder layers, can be trained with two losses at once: a supervised loss on the dataset's labeled categories and a pseudo-label loss derived from Cluster-CLIP's predictions over ImageNet-21K classes. On COCO, LVIS, and nuImages this combination raises mAP@50 by up to 8 points over the previous state of the art on novel categories, with 50 embeddings per image and no CPU clustering at inference. The same pseudo-label branch lets the method work in a semi-supervised setting, where only 1-10% of images are labeled.","feed_headline":"Fine-tuned CLIP head lifts object retrieval by up to 8 points","feed_subtitle":"Learnable queries plus pseudo-labels keep open-vocabulary search accurate with only 50 vectors per image.","key_machinery":"The central object is SUM-CLIP, a variant of CLIP's last attention layer that produces $N$ representative embeddings per image (typically 50) instead of one global embedding or hundreds of dense patch embeddings. Its learnable queries $\\mathbf{Q} \\in \\mathbb{R}^{N \\times C_e}$ are modulated by DETR-style decoder layers conditioned on the image feature map, and then fed into CLIP's original cross-attention with the query, key, value, and output linear layers frozen. This lets gradients flow into a compact set of object summaries while preserving the original CLIP text-image alignment. The second load-bearing mechanism is the two-loss training scheme: the supervised loss uses base-category text embeddings and Hungarian matching (the DETR set-prediction loss), while the pseudo-label loss uses softmax-thresholded Cluster-CLIP similarities to ImageNet-21K text embeddings as targets, extending the vocabulary beyond the dataset's closed set.","core_discovery":"The paper's central claim is that open-vocabulary forgetting, the usual failure mode when fine-tuning CLIP on closed-set labels, can be avoided by training a decoder-style CLIP head with two objectives: one that matches a set of learnable query embeddings to base-category text embeddings, and one that matches the same embeddings to pseudo-labels produced by Cluster-CLIP over ImageNet-21K. The SUM-CLIP head replaces CLIP's final pooling attention with a small set of learnable queries modulated by decoder layers, so the model can summarize multiple objects in an image in a fixed number of embeddings while keeping the last attention layer's linear projections frozen to preserve CLIP's vision-language association. During training a frozen Cluster-CLIP head supplies the pseudo-labels; at inference only the SUM-CLIP head is used. The paper reports that the full FOR system surpasses Cluster-CLIP by up to 8 mAP@50 points on novel categories across COCO, LVIS, and nuImages, and that the pseudo-label branch alone already beats Cluster-CLIP on most benchmarks.","pith_inferences":["Editorial extension: The success of the pseudo-label branch suggests that a similar recipe could improve retrieval for other dual-encoder vision-language models, provided their last attention layers can be converted into a learnable-query head with frozen projections.","Editorial extension: Since the paper acknowledges pseudo-label noise but never quantifies it, a natural next test is measuring how novel-category gains vary with the softmax threshold in Eq. (4); if performance is flat over a wide threshold range, the method is robust to pseudo-label noise, whereas a sharp peak would make threshold selection load-bearing.","Editorial extension: FOR's 50-embedding representation could serve as a compact region-proposal-free feature for open-vocabulary detection or segmentation, trading some spatial precision for a much smaller index footprint; the paper only gestures at these applications."],"forward_implications":["If FOR's central claim holds, object-centric retrieval can be fine-tuned to a target domain without sacrificing open-vocabulary performance, meaning closed-set labels can improve retrieval for queries that were never annotated.","Eliminating CPU clustering at inference, with 50 embeddings per image instead of thousands, makes the method compatible with large-scale nearest-neighbor indexes while roughly tripling visual inference speed relative to Cluster-CLIP.","Because the pseudo-label branch extends beyond dataset categories, the same framework can exploit unlabeled data: at 1-10% labeled data on COCO, novel-category retrieval stays near the fully supervised level.","The method also transfers to open-vocabulary multi-label classification, where it outperforms CLIP and a prior CLIP-based method on COCO-2014.","The design suggests that freezing the linear layers of CLIP's last attention module is a generally useful stability-preserving choice when fine-tuning CLIP heads for retrieval."],"supporting_citations":[{"why":"Supplies the pretrained CLIP vision-language model whose backbone and text encoder FOR builds on and whose last attention layer SUM-CLIP modifies.","marker":"[53]"},{"why":"Defines Cluster-CLIP, the state-of-the-art baseline FOR compares against and the source of pseudo-labels; also supplies the evaluation protocol and the clustering head SUM-CLIP replaces.","marker":"[38]"},{"why":"Introduces Dense-CLIP, the dense-embedding variant of CLIP's head that SUM-CLIP extends with learnable queries and decoder layers.","marker":"[78]"},{"why":"Provides the DETR set-prediction loss with Hungarian matching that FOR adapts for matching image embeddings to base-category and pseudo-label targets.","marker":"[9]"},{"why":"Supplies ImageNet-21K classes used as the pseudo-label vocabulary for open-vocabulary knowledge transfer.","marker":"[13]"},{"why":"Defines the COCO base/novel category split used for evaluation, following the COCO-OVD benchmark convention.","marker":"[5]"},{"why":"Establishes the LVIS base/novel split convention, where rare categories are treated as novel, as used in the experiments.","marker":"[48]"},{"why":"Provides nuImages, one of the three evaluation datasets for which FOR reports retrieval results.","marker":"[7]"}],"fun_headline_variants":["Fine-tuned CLIP head gains 8 mAP on open-vocab retrieval","Forget-free fine-tuning boosts CLIP retrieval by 8 points","8-point jump for open-vocab image retrieval via tuned CLIP","Closed-set fine-tuning preserves open vocab, adds 8 mAP"],"cache_read_input_tokens":23936,"weakest_assumption_plain":"The extra training labels are machine-generated guesses about which of thousands of object categories appear in each image; the whole method depends on those guesses being correct often enough to teach the model about categories it was never given human labels for.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned CLIP head gains 8 mAP on open-vocab retrieval","Forget-free fine-tuning boosts CLIP retrieval by 8 points","8-point jump for open-vocab image retrieval via tuned CLIP","Closed-set fine-tuning preserves open vocab, adds 8 mAP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1316,"prompt_tokens":940,"completion_tokens":376,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":296}},"tokens_in":556,"tokens_out":376,"duration_ms":3611,"temperature":1.0,"reasoning_tokens":296,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:26:22.858928+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Annotate a random sample of the pseudo-labels accepted at the $5 \\times 10^{-4}$ threshold and measure their precision; then retrain FOR with those labels removed or corrected. If the novel-category mAP@50 gain over Cluster-CLIP survives the correction, or vanishes when the labels are removed, the paper's proposed mechanism—that pseudo-label knowledge transfer drives the gain—would be supported or refuted.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained CLIP vision-language model whose backbone and text encoder FOR builds on and whose last attention layer SUM-CLIP modifies."},{"cited_title":"Object- centric open-vocabulary image retrieval with aggregated fea- tures","cited_arxiv_id":null,"evidence_quote":"Defines Cluster-CLIP, the state-of-the-art baseline FOR compares against and the source of pseudo-labels; also supplies the evaluation protocol and the clustering head SUM-CLIP replaces."},{"cited_title":"Extract free dense labels from CLIP","cited_arxiv_id":null,"evidence_quote":"Introduces Dense-CLIP, the dense-embedding variant of CLIP's head that SUM-CLIP extends with learnable queries and decoder layers."},{"cited_title":"End-to- end object detection with transformers","cited_arxiv_id":null,"evidence_quote":"Provides the DETR set-prediction loss with Hungarian matching that FOR adapts for matching image embeddings to base-category and pseudo-label targets."},{"cited_title":"Imagenet: A large-scale hierarchical image database","cited_arxiv_id":null,"evidence_quote":"Supplies ImageNet-21K classes used as the pseudo-label vocabulary for open-vocabulary knowledge transfer."},{"cited_title":"Zero-shot object detection","cited_arxiv_id":null,"evidence_quote":"Defines the COCO base/novel category split used for evaluation, following the COCO-OVD benchmark convention."},{"cited_title":"Simple open-vocabulary object detection with vi- sion transformers","cited_arxiv_id":null,"evidence_quote":"Establishes the LVIS base/novel split convention, where rare categories are treated as novel, as used in the experiments."},{"cited_title":"Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom","cited_arxiv_id":null,"evidence_quote":"Provides nuImages, one of the three evaluation datasets for which FOR reports retrieval results."}],"review_version":1}