{"id":"343506ef-55f4-4100-9b90-c6c7b4d53dba","arxiv_id":"2507.21489","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A CLIP plus MLLM pipeline with additive-bias Low-Rank adaptation retrieves 3D objects of unseen categories from multi-view images, outperforming prior art by about 10% mAP on average.","lead":"This paper presents DAC, a pipeline for open-set 3D object retrieval that combines CLIP with a multimodal language model that writes text descriptions of objects. The method beats previous approaches on four benchmarks using only multi-view images, while prior work used point clouds, voxels, and even test data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-dataset fusion weight α appears to be selected on the test set, so the reported +10.01% mAP gain may be inflated by test-set fitting.","rationale":"The paper presents a sensible method and thorough ablations, and the main empirical claim is plausible if the evaluation is clean. However, the selection of the fusion weight α is the most load-bearing assumption: Table 8 explicitly labels per-dataset α values as \"optimal\" and sets them as defaults, with no description of a validation protocol. Since α directly controls the contribution of MLLM text features in Eq. (6) and differs per dataset, tuning it on the test set would inflate the reported gains, especially on OS-NTU-core where the margin is large. The reader's weakest_assumption identifies exactly this issue, and I concur. The minor arithmetic discrepancy between the computed average improvement (≈9.79% mAP for ViT-L/14 over HGM2R) and the stated +10.01% is secondary but reinforces the need for a precise, reproducible evaluation protocol. I therefore recommend keeping the reader's CONDITIONAL verdict and adding the concrete validation test above.","tokens_in":24227,"tokens_out":6768,"duration_ms":80205,"concrete_test":"For each dataset, select α on a held-out validation split drawn from the seen training categories (or use one fixed α, e.g., 0.4, across all datasets), then recompute the open-set DAC rows in Table 1. If the average mAP improvement over HGM2R drops below the claimed +10.01%, or if the OS-NTU-core margin of +20.95 mAP shrinks materially, the reported gains are inflated by test-set hyperparameter selection. Report the α values chosen and the resulting mAP for both ViT-B/32 and ViT-L/14.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract: \"DAC significantly surpasses prior arts by an average of +10.01% mAP\") rests on the open-set rows of Table 1, and those numbers depend on per-dataset fusion weights α in Eq. (6). Appendix B.2 reports \"optimal α values across all datasets and backbones\" (Table 8) and states \"We set these values as our default configurations.\" No held-out validation split is described for selecting α; the word \"optimal\" strongly suggests the values were chosen by inspecting test-set performance. This is a small but consequential degree of freedom: four datasets, four α values, and the largest single gain (OS-NTU-core ViT-L/14, +20.95 mAP over HGM2R) could be substantially reduced if α were fixed or chosen on a validation split. The paper also reports no error bars or multiple seeds, so the stability of the claimed improvements is unknown. If α was tuned on the test set, the headline improvement is an optimistic estimate of generalization. This is load-bearing because the entire state-of-the-art claim is quantitative, and the protocol for the most important hyperparameter is not specified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DAC, a framework for open-set 3D object retrieval that uses only multi-view images. DAC first employs an off-the-shelf MLLM (InternVL) to generate category-level descriptions for training and object-level descriptions at inference, then adapts a frozen CLIP encoder with a proposed Additive-Bias Low-Rank Adaptation (AB-LoRA), and finally combines visual and textual CLIP embeddings via the weighted fusion h = tanh(g + α f_t). Experiments on four open-set 3DOR datasets report an average improvement of +10.01% mAP over prior art in the open-set setup, together with cross-dataset, single-image, and depth-image retrieval experiments.","tokens_in":24471,"tokens_out":5929,"duration_ms":66669,"significance":"If the reported numbers hold, DAC is a simple and effective new state of the art for open-set 3DOR, notable for relying only on multi-view images and for not using test data during training, in contrast to HGM2R. The paper is thorough in its ablations (LoRA rank, fusion scheme, normalization, prompts, MLLM choice, view numbers), releases code, and explicitly discusses limitations in Sec. 4.6. The central caveat is the evaluation protocol for the per-dataset fusion weight α and the absence of variance estimates; because the headline gain is purely quantitative, these issues directly affect the strength of the state-of-the-art claim.","major_comments":[{"comment":"The fusion weight α in Eq. (6) is reported as an 'optimal' value per dataset and per backbone in Table 8, with the text stating 'We set these values as our default configurations.' No held-out validation split is described for choosing these eight values, and Figure 5 sweeps α on OS-MN40-core and reports the resulting test-set mAP. Consequently, the headline improvement of +10.01% mAP in the Abstract, which rests on the open-set rows of Table 1, may be inflated by test-set fitting. The concern is concrete: the largest single gain (OS-NTU-core, ViT-L/14, +20.95 mAP over HGM2R) uses α=0.3, and the cross-dataset and single-image results in Tables 5 and 6 also rely on per-dataset α choices. Please either specify a validation-based selection protocol that fixes α independently of the test set, or report sensitivity with a fixed α across all datasets and show that the ranking over prior methods is preserved.","section":"Appendix B.2 (Table 8) and Eq. (6)"},{"comment":"All retrieval results are reported as single runs without error bars or multiple seeds. The Abstract uses the word 'significantly surpasses,' but with one run per configuration there is no estimate of variance, and the claimed margins (e.g., +12.31% mAP in Table 5) could be within run-to-run variability. Please provide mean and standard deviation over at least three independent training runs for the main comparisons, or at least for Table 1 and Table 8, and state the number of seeds used for every reported number.","section":"Tables 1, 5, and 6"}],"minor_comments":[{"comment":"There are several typos: 'pretrainned' should be 'pretrained', 'Multi-model Large Language Models' should be 'Multi-modal', 'siginantly' should be 'significantly', and 'remarkablely' should be 'remarkably'.","section":"Sec. 3.1 and Sec. 4.2"},{"comment":"Figures 5 and 8 appear in the submitted PDF as raw embedded paths (e.g., '/uni00000013/...') rather than rendered plots; the figures need to be regenerated before publication.","section":"Appendix B.2 and Appendix G"},{"comment":"Please clarify whether α is applied after normalizing g and f_t individually, and state the search grid and step size used to arrive at the 'optimal' α values in Table 8.","section":"Eq. (6) and Table 8"},{"comment":"The 'Zero-shot Setup' should be defined explicitly; the DAC rows in this block use MLLM-generated descriptions of the query object, so the setup is not identical to prior zero-shot methods that do not use query-specific text.","section":"Table 1 and Sec. 4.2"},{"comment":"The gradient derivation and the statement that adding Φ 'breaks the tight linkage' is heuristic; Table 3 provides empirical support, but the text should not imply a formal proof of why the bias improves unseen-category generalization.","section":"Sec. 3.2"},{"comment":"The new ZS-Objaverse-Core benchmark should specify the number of categories and the exact 20%/80% split protocol more precisely, since current wording only says 'each category' without stating the number of categories used.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-executed and the central idea is attractive, but the per-dataset 'optimal' α values in Table 8 are a genuine protocol concern. If α was selected by peeking at the test set, the reported margins may be overestimates; the authors should be asked directly and, if necessary, the main tables should be recomputed with a validation-based protocol. The missing error bars compound this issue. The supplementary PDF also has rendering problems that should be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely simple and mostly effective recipe for open-set 3D retrieval — use an MLLM to generate text descriptions, adapt CLIP with a biased LoRA, then add the text and image features. The ablations are thorough and the code is out. But the paper's headline number, +10.01% mAP, does not match the numbers in Table 1, and the way α is chosen per dataset in Table 8 looks like test-set tuning. Those two issues make the SOTA claim weaker than it appears.\n\nWhat's new and good: combining CLIP with an MLLM for open-set 3DOR, where the MLLM provides both training-time category descriptions and inference-time object descriptions, is a sensible contribution. The additive-bias LoRA variant is simple and consistently helps over plain LoRA in their ablation. The experiments cover multiple datasets, cross-dataset transfer, single-image retrieval, and even depth-based point cloud retrieval. The paper is transparent about limitations. This is a solid empirical chapter, not a breakthrough.\n\nWhere it's soft: First, the α values in Table 8 are called 'optimal' and are set per dataset and backbone. No validation protocol is described. Given that α strongly affects performance (Figure 5), and that the largest per-dataset gain (+20.95 mAP on OS-NTU-core with ViT-L/14) uses α=0.3, it's hard to avoid the conclusion that α was chosen by peeking at the test set. This inflates the reported gains. Second, there are no error bars or multiple seeds; with training sets as small as 98 objects, single-run results are fragile. Third, the 'Zero-shot Setup' in Table 1 includes DAC, which is fine-tuned on the training set, while the other methods in that block are not. That comparison is unfair and should be relabeled or removed. Fourth, the abstract's +10.01% mAP average doesn't reproduce from Table 1: against HGM2R in the open-set rows, the average gains are about +5.6% (ViT-B/32) and +9.8% (ViT-L/14). The paper should show exactly how that number is computed.\n\nNone of these are fatal. The method's general direction is sound and the core idea is likely to work with a properly chosen α (e.g., fixed or tuned on a real validation split). A serious reviewer should ask for (a) a clear hyperparameter selection protocol, (b) multi-seed variance, (c) a corrected comparison table, and (d) a re-check of the headline number.\n\nMy recommendation: send it to review, but expect major revisions on evaluation honesty. It's a useful paper for the 3D retrieval / CLIP adaptation community, and the AB-LoRA trick is worth citing.","headline":"Good simple idea and thorough ablations, but the headline mAP gain doesn't match Table 1 and the per-dataset α looks test-set tuned.","tokens_in":24968,"tokens_out":5086,"would_cite":true,"duration_ms":51221,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Unseen 3D object retrieval jumps ~10 mAP with CLIP plus text","keywords":["open-set 3D object retrieval","CLIP","multi-modal large language model","low-rank adaptation","multi-view images","zero-shot retrieval","cross-dataset retrieval"],"falsifier":"Re-run the main comparison with a single fixed $\\alpha$ for all four datasets, or choose $\\alpha$ on a held-out validation split drawn from the probe/gallery distribution before evaluation; if the average margin over HGM2R falls below the claimed 10.01 mAP, the headline result depends on per-benchmark test-set tuning.","tokens_in":24066,"feed_emoji":"🔍","tokens_out":5837,"duration_ms":58789,"temperature":0.7,"pith_summary":"Open-set 3D object retrieval asks a system to fetch 3D models of categories it never saw during training, and most prior methods need voxels, point clouds, and test-data supervision to do this. This paper proposes DAC, which uses only multi-view images, a frozen CLIP encoder, and an off-the-shelf multimodal language model that writes one-sentence descriptions of the seen categories during training and of each query object at inference. A lightweight additive-bias LoRA adapts CLIP to the rendered views, and the visual and textual embeddings are combined by weighted addition followed by tanh normalization. The authors report that DAC beats the previous best method by an average of 10.01 mAP across four open-set 3DOR benchmarks, and that it also improves cross-dataset, single-image, and depth-based retrieval.","feed_headline":"Unseen 3D object retrieval jumps ~10 mAP with CLIP plus text","feed_subtitle":"Training on images only, plus language-model descriptions, beats multi-modal rivals on four benchmarks.","key_machinery":"The load-bearing mechanism is the pairing of CLIP's aligned image-text embedding space with an MLLM that writes descriptions, plus a modified parameter-efficient adapter. AB-LoRA extends LoRA by adding a trainable bias vector $\\Phi$ to each adapted linear layer's output, $o = W_o z + \\gamma B A z + \\Phi$; because LoRA's update accumulates information from seen-category inputs, the bias is intended to prevent that update from overfitting the seen categories and to preserve generalization to unseen ones. During training the MLLM generates one description per category, which becomes the text classifier in a cross-entropy contrastive loss over pooled multi-view features; at inference it generates one description per object, whose CLIP text embedding is added to the visual embedding with weight $\\alpha$ and passed through tanh to form the final 3D descriptor.","core_discovery":"On the paper's own terms, the discovery is that a generative multimodal large language model and a discriminative contrastive model can be combined into a strong open-set 3D retriever without any 3D backbone and without touching test data. The method works in three steps: an MLLM supplies category-level descriptions that align with CLIP's training objective, an Additive-Bias Low-Rank adaptation (AB-LoRA) fine-tunes the attention weights of CLIP's visual and textual encoders to close the gap between natural images and multi-view projections while resisting overfitting to seen classes, and a simple elementwise fusion $h = \\tanh(g + \\alpha f_t)$ joins pooled multi-view features with text embeddings. Trained only on seen categories and evaluated on unseen ones, DAC reports 58.70 to 70.74 mAP across the four benchmarks with a ViT-L/14 backbone, surpassing HGM2R and other multi-modal baselines while using considerably less input information.","pith_inferences":["If $\\alpha$ were chosen by a principled rule, such as predicting it from the MLLM's confidence or fixing it across all four benchmarks, the method would be more likely to transfer to a new dataset without per-benchmark tuning; the current table of optimal $\\alpha$ values leaves that question open.","The same Describe-Adapt-Combine pattern could be applied to other retrieval domains where CLIP has a strong prior and an MLLM can verbalize unseen classes, such as sketch retrieval or fine-grained species identification, though the paper does not test these.","Because stronger MLLMs (up to Qwen2.5-VL-7B) monotonically improve DAC's mAP, part of the claimed gap over prior methods is a bet on continuing MLLM progress rather than on the 3D representation itself.","A testable extension: replacing the scalar $\\alpha$ with a per-query weight, based on how confident the MLLM's description is, could improve retrieval on hard pairs like wardrobe versus bookshelf that currently fail."],"forward_implications":["A 3D retriever can be built without point-cloud or voxel encoders: projecting an object into 24 images and reading its category descriptions is enough to beat multi-modal baselines.","The same recipe transfers to cross-dataset retrieval, single-image retrieval, and zero-shot depth-image retrieval, suggesting the CLIP-plus-MLLM synergy is not limited to one dataset or modality.","The additive bias in AB-LoRA is what lets the model keep generalization to unseen categories; removing it costs about 2.55 mAP on OS-MN40-core.","If the per-dataset fusion weight $\\alpha$ is a genuine design choice rather than a test-set fit, DAC's margin over HGM2R holds without any test data during training, which makes the method practical for repositories that grow over time."],"supporting_citations":[{"why":"Supplies the four open-set 3DOR benchmarks, the 24-view rendering scheme, and HGM2R, the previous state of the art that DAC is compared against.","marker":"[18]"},{"why":"Provides the frozen CLIP ViT encoders whose aligned image-text space DAC adapts with AB-LoRA and fuses text into.","marker":"[50]"},{"why":"InternVL is the MLLM used to generate category descriptions during training and object descriptions at inference; the paper also scales it from 1B to 8B variants.","marker":"[9]"},{"why":"LoRA is the parameter-efficient low-rank adaptation method that AB-LoRA extends with an additive bias to improve unseen-category generalization.","marker":"[27]"},{"why":"Defines the SHREC'22 open-set 3DOR task and its difficulties, the problem statement DAC is designed to solve.","marker":"[17]"},{"why":"MV-CLIP is the zero-shot multi-view CLIP baseline re-implemented for comparison; it requires ground-truth category sets for view selection, unlike DAC.","marker":"[53]"},{"why":"ULIP-2 is a strong point-cloud zero-shot baseline used for comparison and for the ZS-Objaverse-Core depth-image retrieval experiment.","marker":"[65]"},{"why":"OpenShape provides zero-shot point-cloud CLIP baselines that DAC surpasses in the depth-image extension.","marker":"[39]"}],"fun_headline_variants":["Text-augmented CLIP lifts open-set 3D retrieval by 10% mAP","CLIP plus MLLM descriptions boosts open-set 3D retrieval mAP by 10","Image-only CLIP + text hints achieve 10% mAP gain for open-set 3D","Unseen 3D retrieval: CLIP adapted with MLLM text beats multi-modal baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains assume the per-dataset fusion weights $\\alpha$ listed as 'optimal' were chosen without looking at test retrieval labels; if they were tuned on the test set, the headline margins, especially the roughly 20 mAP gain on OS-NTU-core, would be inflated by test-set fitting.","fun_headline_variants_meta":{"raw":{"variants":["Text-augmented CLIP lifts open-set 3D retrieval by 10% mAP","CLIP plus MLLM descriptions boosts open-set 3D retrieval mAP by 10","Image-only CLIP + text hints achieve 10% mAP gain for open-set 3D","Unseen 3D retrieval: CLIP adapted with MLLM text beats multi-modal baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000731,"raw_usage":{"total_tokens":3322,"prompt_tokens":1044,"completion_tokens":2278,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":2177}},"tokens_in":660,"tokens_out":2278,"duration_ms":19785,"temperature":1.0,"reasoning_tokens":2177,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:41:22.772955+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the main comparison with a single fixed $\\alpha$ for all four datasets, or choose $\\alpha$ on a held-out validation split drawn from the probe/gallery distribution before evaluation; if the average margin over HGM2R falls below the claimed 10.01 mAP, the headline result depends on per-benchmark test-set tuning.","supporting_citations":[{"cited_title":"Hypergraph-based multi-modal represen- tation for open-set 3d object retrieval.IEEE TPAMI, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the four open-set 3DOR benchmarks, the 24-view rendering scheme, and HGM2R, the previous state of the art that DAC is compared against."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"Provides the frozen CLIP ViT encoders whose aligned image-text space DAC adapts with AB-LoRA and fuses text into."},{"cited_title":"Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks","cited_arxiv_id":null,"evidence_quote":"InternVL is the MLLM used to generate category descriptions during training and object descriptions at inference; the paper also scales it from 1B to 8B variants."},{"cited_title":"LoRA: Low-rank adaptation of large language models","cited_arxiv_id":null,"evidence_quote":"LoRA is the parameter-efficient low-rank adaptation method that AB-LoRA extends with an additive bias to improve unseen-category generalization."},{"cited_title":"Shrec’22 track: Open-set 3d object retrieval.Computers & Graphics, 107:231–240, 2022","cited_arxiv_id":null,"evidence_quote":"Defines the SHREC'22 open-set 3DOR task and its difficulties, the problem statement DAC is designed to solve."},{"cited_title":"Ulip-2: Towards scal- able multimodal pre-training for 3d understanding","cited_arxiv_id":null,"evidence_quote":"ULIP-2 is a strong point-cloud zero-shot baseline used for comparison and for the ZS-Objaverse-Core depth-image retrieval experiment."},{"cited_title":"Openshape: Scaling up 3d shape representation towards open-world understanding.NeurIPS, 36, 2023","cited_arxiv_id":null,"evidence_quote":"OpenShape provides zero-shot point-cloud CLIP baselines that DAC surpasses in the depth-image extension."}],"review_version":1}