{"id":"9305756d-6823-4a41-8ec5-e7d84281e13b","arxiv_id":"2608.01794","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"By adding identity-aware sampling and a contrastive loss on a new 28-dataset benchmark, the authors build multimodal embeddings that are far better at visual identity matching without losing general retrieval accuracy.","lead":"New identity-aware training makes universal multimodal embedding models much better at matching images of the same person, product, or landmark, while keeping their general retrieval skills. The paper adds a large benchmark, MVEB, covering four identity tasks, and shows large gains over existing embedding models on it.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Random-split fallback in MVEB curation may leak training identities into evaluation, threatening the out-of-domain claim.","rationale":"The paper makes a solid empirical case that identity-aware sampling and contrastive training improve identity-centric retrieval in UMEs, and the candidate-pool scaling analysis in Table 7 adds robustness evidence. However, the out-of-domain generalization claim is the crux: the authors state that identity-disjoint splits are enforced only 'whenever feasible,' and explicitly admit a random-split fallback for datasets with insufficient identities. Because the paper never identifies which sub-datasets fall under this fallback, the reader cannot verify that the large MVEB gains reflect unseen-identity generalization rather than label memorization. This is not an accusation of misconduct; it is an unaddressed methodological ambiguity in a self-constructed benchmark. The proposed test—quantifying identity overlap and re-evaluating on a strict subset—would settle the concern. If the gains persist, the paper's claim stands; if not, the central claim is overstated. I therefore keep the reader's CONDITIONAL verdict unchanged, with the added condition that split metadata be released and the strict-split re-evaluation be reported.","tokens_in":21548,"tokens_out":5631,"duration_ms":62524,"concrete_test":"Release or reconstruct the MVEB split metadata; for each OOD sub-dataset, compute the Jaccard overlap between identity labels in the MVEB training split and identity labels in the evaluation split. Then re-run Table 6 on the subset of OOD datasets with provably zero identity overlap, re-splitting any dataset that fell back to random splits using external identity annotations from the source datasets. If VisME's OOD MVEB margin over the strongest baseline shrinks by more than ~5 points on this strict subset, the out-of-domain claim is compromised; if the margin persists, leakage is not the main driver.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 Step 3 states: 'Crucially, whenever feasible, we enforce a strict identity-aware split: no identity present in training appears in any evaluation set. For sub-datasets with insufficient unique identities, we fall back to a standard random split at the sample level.' The central empirical claim — that VisME generalizes to unseen identities and outperforms UME baselines by 25–30 points on MVEB — depends on evaluation identities being disjoint from training identities, especially for the 8 OOD sub-datasets. The random-split fallback breaks this condition for any sub-dataset where it was invoked: test identities may have appeared in training, so MVEB scores could reflect memorization of identity labels rather than identity discrimination. The paper does not disclose which sub-datasets used the fallback, nor the extent of identity overlap. The supplementary statement that 'there is no overlap of specific images between the training and evaluation sets' addresses image duplication, not identity leakage. Since the benchmark is self-constructed and the model is trained on 20 of the same datasets, this unquantified leakage is the most direct threat to the headline result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that universal multimodal embedding (UME) models neglect visual identity discrimination (VisID) and proposes to close this gap in three parts: a formal decomposition of VisID into four meta-tasks (Identity Recognition, Re-Identification, Identity Grounding, Identity Editing); a new benchmark, MVEB, curated from 28 real and synthetic datasets, with 20 training and 8 out-of-domain (OOD) evaluation sets; and a training framework, VisME, that augments standard UME contrastive training with identity-aware offline sampling and structured hard negatives. Experiments on Qwen2-VL/Qwen2.5-VL backbones show large MVEB gains over existing UMEs (e.g., MVEB average 78.8 for the 7B model vs. 52.3 for VLM2Vec-7B) while roughly matching MMEB performance. Code and data are promised for release.","tokens_in":21858,"tokens_out":6293,"duration_ms":71370,"significance":"If the results hold, this is a useful contribution. VisID is indeed underexplored in the UME literature, and the proposed four-task decomposition is a reasonable organizing principle. MVEB, despite being author-constructed, is larger and more identity-focused than existing UME benchmarks and would be a valuable community resource if released with clean split metadata. The method itself is simple and compatible with existing UME pipelines, and the reported gains on identity-centric tasks are substantial. The external MMEB evaluation and the OOD splits provide some check against pure overfitting to the proposed benchmark. However, the strength of the central claim is weakened by a self-constructed benchmark with an admitted identity-leakage fallback, by single-run comparisons with no error bars, and by a definition of 'identity' that mixes instance-level and category-level labels.","major_comments":[{"comment":"The benchmark's identity-aware split has an exception that directly affects the headline numbers. The text states: 'Crucially, whenever feasible, we enforce a strict identity-aware split... For sub-datasets with insufficient unique identities, we fall back to a standard random split at the sample level.' Table 6 marks Cars196 and iNat as 'ID-Unseen? = ✗', and both are also training sub-datasets (Figure 3). Thus the reported 'All IND' and overall MVEB averages include in-domain evaluations on identities that may have appeared in training. The supplementary statement that 'there is no overlap of specific images between the training and evaluation sets' does not address identity overlap. I acknowledge that the eight OOD rows are all marked 'ID-Unseen? = ✔', so the OOD average may be less affected; the problem is the overall/IND scores and the general claim of generalizing to unseen identiti","section":"Sec. 4.1 Step 3, Table 6, Supp. C"},{"comment":"All experiments appear to be single runs with no error bars or significance tests. This matters most for the MMEB claim: VisME (Qwen2.5-VL-7B) scores 72.2 vs. B3's 72.0, a 0.2-point difference, and the text states 'Our method establishes best results on both MMEB and MVEB benchmarks.' With a single run, a 0.2-point gap is not distinguishable from training noise. Similarly, the temperature ablations in Table 5 differ by only 0.1–0.5 points, and the interleave batch size comparison in Table 4 shows a 0.1-point difference at the top end. Please provide means and standard deviations over at least three seeds for the main results and for the comparisons where the difference is small, or perform a paired significance test. The large MVEB margins are probably robust, but the MMEB 'best' claim is not supported without this.","section":"Tables 1, 2, 4, 5"},{"comment":"The formulation defines identity as 'either a unique visual instance or a fine-grained semantic category', and the benchmark accordingly treats iNat species labels and GLDv2 landmark labels as identities alongside face and vehicle identities. This conflation is materially relevant to the paper's motivation: instance retrieval, person re-identification, and identity preservation in AIGC are about unique entities. A model that improves on 'same species' or 'same landmark' may be learning fine-grained category recognition rather than instance-level identity discrimination. Please report the MVEB results separately for instance-level sub-datasets (unique objects, faces, persons, vehicles, products) and category-level sub-datasets (iNat, GLDv2, possibly MET), and discuss whether the gains differ. This does not invalidate the benchmark, but it is necessary for interpreting what VisME actually","section":"Sec. 3, Tables 8 and 9"}],"minor_comments":[{"comment":"The section heading reads 'Methology'; should be 'Methodology'.","section":"Sec. 4 heading"},{"comment":"There is a duplicated phrase: 'optimized under a single, consistent objective objective(q,C+,C−)'.","section":"Sec. 4.2.1"},{"comment":"The notation 'C − i )' has an unbalanced parenthesis; the negative set should be written consistently as C−_i.","section":"Eq. (4)"},{"comment":"The adaptation protocol for dual-encoder models is ad hoc: all tasks except Identity Editing are converted to pure image-to-image by dropping text, and Identity Editing uses averaged visual and text embeddings. This is not the native task format for these models and may understate their performance. Please describe the protocol more precisely, justify it, or move dual-encoder results to a clearly labeled 'adapted protocol' subsection. The central UME comparison is unaffected, but the current Table 1 mixes native and adapted evaluations.","section":"Sec. 5.1 / 5.2"},{"comment":"The 'ID-Unseen?' column is informative but is only defined in the caption. Please add a sentence in the main text explaining that this column encodes whether the identity-aware split was applied, and note explicitly in the caption that '✗' indicates the random-split fallback.","section":"Table 6"},{"comment":"The baseline name is inconsistently written as 'VLM2VEC' here; elsewhere it is VLM2Vec.","section":"Supp. B"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is positioned for a top-tier venue. The main empirical result is likely correct in direction, but the current presentation overstates the strength of the evidence in two ways: the MMEB 'best' claim rests on a 0.2-point single-run gap, and the identity-split fallback is disclosed only in passing. I would encourage the editor to require the authors to release the identity-split metadata and per-sub-dataset overlap counts as a condition of publication, and to ask for multi-seed results on the main comparisons. The paper is worth pursuing; the fixes are achievable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something real: it gives UMEs a concrete formulation of visual identity discrimination, builds a 28-dataset benchmark to measure it, and shows that a simple identity-aware batch sampling scheme produces large improvements over strong baselines. The MVEB numbers are impressive, and the ablations make a convincing case that the gains come from the proposed sampling rather than from the extra training data. I also liked the candidate-pool scaling analysis and the fact that they check general performance on MMEB rather than only on their own benchmark. The soft spots are real but not evenly weighted. The biggest one is the random-split fallback in MVEB curation. The paper states in Section 4.1 that whenever feasible they enforce a strict identity-aware split, but for datasets with insufficient unique identities they fall back to a sample-level random split. The paper never says which sub-datasets used the fallback or how many identities leak. That is a transparency problem. The OOD claim, that VisME generalizes to unseen identities, is weakened wherever the fallback was used, because an identity seen in training can appear in evaluation. I do not think this sinks the paper, because the gains are huge and consistent across many datasets, and the MMEB results provide an independent sanity check. But the authors need to disclose the fallback usage and, ideally, re-report results excluding any leaked test identities. The other issues are more minor. Single-run results without error bars are common in this area but still a limitation. The benchmark is self-constructed, which is a circularity concern, though the MMEB evaluation helps. And the paper omits IDMR, a directly related method, from the baseline comparison, a strange choice given that IDMR datasets are included in MVEB. That should be fixed in revision. Overall, the central empirical claim holds up as far as I can tell. The paper deserves serious peer review, but the reviewers should push for disclosure of the split fallback and a reanalysis on the clean subset. If the authors provide that, this becomes a solid contribution to the multimodal-embedding subfield.","headline":"A genuinely useful benchmark and training recipe for identity-aware UMEs, but the undisclosed random-split fallback muddies the out-of-domain claim and needs to be addressed before I'd trust the numbers.","tokens_in":605,"tokens_out":670,"would_cite":true,"duration_ms":33201,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Visual identity discrimination, long missing from universal multimodal embeddings, can be added through identity-aware sampling and a unified contrastive loss, yielding ~25-point gains on a new MVEB benchmark without hurting general perform","keywords":["visual identity discrimination","universal multimodal embeddings","contrastive learning","identity-aware sampling","multimodal retrieval","benchmark","re-identification","identity-preserving generation"],"falsifier":"Train and evaluate on a version of MVEB where identity is defined strictly at instance level (each identity is one unique object/person/face, not a category), and where every sub-dataset uses a fully identity-disjoint split with no identity overlap between train and test. If the ~25-point advantage of VisME over baselines shrinks substantially or vanishes, the central claim would be shown to depend on category-level labels and identity leakage rather than on true identity discrimination.","tokens_in":21517,"feed_emoji":"🎯","tokens_out":5901,"duration_ms":58007,"temperature":0.7,"pith_summary":"This paper argues that universal multimodal embedding (UME) models, which align images and text in one space, have silently neglected visual identity discrimination: telling whether two images show the same person, product, landmark, or edited subject. The authors formalize this as VisID, split into four meta-tasks (recognition, re-identification, grounding, editing), and build MVEB, a 28-dataset benchmark with 522K samples to train and test it. They then show that a simple add-on—offline identity-aware batch sampling plus structured hard negatives under one contrastive loss, called VisME—raises MVEB scores by roughly 25–30 points over the best prior UMEs while matching or slightly beating them on the general MMEB benchmark. If correct, this means identity-level retrieval can be a first-class capability of general-purpose multimodal embeddings, with direct use in instance retrieval, person/vehicle re-ID, and identity-preserving AI generation.","feed_headline":"Identity-aware training lifts multimodal embeddings by ~25 points","feed_subtitle":"New MVEB benchmark and VisME model add fine-grained identity retrieval without losing general performance.","key_machinery":"The load-bearing mechanism is offline identity-aware batch pre-scheduling: all mini-batches are generated before training so each identity appears at most once per batch, with sampling probability proportional to identity size, eliminating false negatives from repeated identities. Structured hard-negative sampling then inserts designated same-semantics/different-identity pairs, and a single unified contrastive loss with scaled cosine similarity (learnable temperature) optimizes both semantic and identity triplets. The MVEB curation pipeline, including identity-disjoint train/test splits, is the measurement machinery that makes the capability visible.","core_discovery":"Visual identity discrimination—matching a query image to other instances of the same person, product, landmark, or edited subject—is treated here as a missing core capability of universal multimodal embeddings. The authors define four meta-tasks (recognition, re-identification, grounding, editing), build MVEB (28 datasets, 522K samples, identity-disjoint splits where feasible), and train VisME with offline identity-aware batch pre-scheduling, structured hard negatives, and a single contrastive loss. VisME's MVEB averages of 69.1 (2B) and 78.8 (7B) exceed the strongest prior UME by roughly 25–30 points while matching MMEB (72.2 vs 72.0), supporting the claim that identity and generality can b","pith_inferences":["If the identity-aware batch pre-scheduling generalizes, the same false-negative-avoidance trick could improve contrastive learning in other settings where the same entity reappears across samples, such as audio speaker embeddings or document embeddings with repeated entities.","MVEB's 'identity' includes fine-grained categories (species, landmarks, product types), so part of the reported gain may be improved fine-grained classification rather than true instance-level discrimination; a strict instance-only split would reveal the size of that gap.","The paper's reliance on an auxiliary embedding model for hard-negative mining suggests a self-improving loop: better identity embeddings would produce better hard negatives, which could in turn train even better identity embeddings.","The supplementary candidate-pool scaling experiment shows that even VisME degrades when the gallery grows from 3k to 100k candidates, so gallery-scale identity retrieval remains an open problem that the planned larger MVEB will measure."],"forward_implications":["UME models trained with the VisME recipe gain a capability that previous universal embeddings lacked: matching a query image to other instances of the same identity under a language instruction.","MVEB supplies a reusable 28-dataset evaluation across four identity meta-tasks, with 8 out-of-domain sets, so future models can be compared on identity discrimination directly.","Identity-aware embeddings of this kind can serve as data-curation tools and evaluation metrics for identity-preserving AI-generated content.","Because identity-aware sampling removes false negatives, training is more stable at large batch sizes and benefits from hard-negative mining, which the ablations show improves both in-distribution and out-of-distribution scores.","Scaling the base MLLM from 3B to 7B parameters consistently improves identity performance, with the largest gains on face datasets, suggesting that larger models capture finer identity cues."],"supporting_citations":[{"why":"Supplies the MMEB benchmark and training corpus, and provides the VLM2Vec baseline that the paper extends and compares against.","marker":"[25]"},{"why":"Supplies the hard-negative batch-mining technique (k=5) and the strong B3 baseline that VisME outperforms.","marker":"[55]"},{"why":"Provides Qwen2.5-VL, the base MLLM for the main VisME models.","marker":"[3]"},{"why":"Provides Qwen2-VL, the base MLLM for the 2B and 7B VisME variants.","marker":"[58]"},{"why":"Source dataset for landmark identity-recognition training samples after identity-aware re-sampling.","marker":"[62]"},{"why":"Source dataset for fine-grained species identity-recognition training and evaluation.","marker":"[57]"},{"why":"AIGC editing dataset used for identity-editing training and hard-negative mining.","marker":"[60]"},{"why":"Provides the auxiliary embedding model used for denoising and hard-negative mining in AIGC data.","marker":"[2]"}],"fun_headline_variants":["VisME gains ~25 points on identity tasks, keeping general performance","Identity-aware embeddings: 25-point jump on MVEB without losing generality","MVEB benchmark helps VisME match general tasks while nailing identity"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The formulation equates identity with the label each source dataset provides, so across datasets identity ranges from a unique instance (a face, a product) to a fine-grained category (a species, a landmark), and for datasets with too few identities the evaluation split is random rather than identity-disjoint, meaning some test identities may already be in training.","fun_headline_variants_meta":{"raw":{"variants":["VisME gains ~25 points on identity tasks, keeping general performance","Identity-aware embeddings: 25-point jump on MVEB without losing generality","MVEB benchmark helps VisME match general tasks while nailing identity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001054,"raw_usage":{"total_tokens":4282,"prompt_tokens":787,"completion_tokens":3495,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":3445}},"tokens_in":531,"tokens_out":3495,"duration_ms":23758,"temperature":1.0,"reasoning_tokens":3445,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T20:37:38.865320+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train and evaluate on a version of MVEB where identity is defined strictly at instance level (each identity is one unique object/person/face, not a category), and where every sub-dataset uses a fully identity-disjoint split with no identity overlap between train and test. If the ~25-point advantage of VisME over baselines shrinks substantially or vanishes, the central claim would be shown to depend on category-level labels and identity leakage rather than on true identity discrimination.","supporting_citations":[],"review_version":1}