{"id":"691f6e25-b835-469b-aa1e-e3a201b48134","arxiv_id":"2508.20188","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning Qwen2-VL to predict 16 quantitative skin attributes yields embeddings that retrieve images matching a query in both appearance and the target attribute.","lead":"This study fine-tunes a multimodal language model on 16 numeric skin-lesion attributes and shows its image embeddings can retrieve similar lesions, including matches on a chosen attribute. It suggests a path toward AI dermatology tools that explain decisions with measurable features instead of black-box scores.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Retrieval evaluation uses the training set as the gallery, so the claimed attribute-grounding advantage may reflect training-set familiarity rather than generalizable embedding geometry.","rationale":"The paper claims that fine-tuning an MLLM to predict quantitative skin attributes grounds its embedding space, with retrieval as the concrete evidence. The reader's weakest assumption—that using the training set as the retrieval database biases the evaluation—is exactly the most load-bearing concern. The fine-tuned model has seen all gallery images during training, so its embeddings for those images are not representative of how it would embed novel images. The baselines lack this advantage, making the comparison unfair. The held-out R2 results are strong and suggest some attribute information is encoded generally, but they do not validate the retrieval-specific claim about the embedding space. The proposed fix—using a held-out gallery—directly tests the generalizability of the grounding and would settle whether the concern lands. I agree with the reader's CONDITIONAL verdict: the paper should be accepted only after the retrieval evaluation is repeated with a held-out database or otherwise controlled for training-set familiarity. My concern does not change the verdict, hence UNCHANGED.","tokens_in":8299,"tokens_out":3312,"duration_ms":38861,"concrete_test":"Re-run the quantitative retrieval experiment (Section 4.2) with a held-out gallery: split the 118,495 test images (stratified by patient) into a gallery of ~94,000 and a query set of ~24,000. Compute the same percentile-rank boxplots for the four Qwen variants and MONET. If the fine-tuned image-text embeddings no longer consistently outperform image-only and the untuned baseline, or the advantage diminishes dramatically, the reported retrieval effect is an artifact of training-set familiarity rather than general attribute grounding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MLLM embedding spaces can be grounded in quantitative attributes is supported primarily by the retrieval case study (Section 3.3 and 4.2). However, the retrieval database is explicitly set to I' = Itr (Section 3.3), and Itr is the same set of images used for LoRA fine-tuning (Section 4.1). The fine-tuned model was optimized to predict attribute values for exactly those images, so its embeddings for Itr may be calibrated to the training-set attribute distribution. This creates an unfair comparison against the vanilla Qwen and MONET baselines, which have never seen these images. The reported superiority of the fine-tuned embeddings—and especially of the attribute-conditioned h(im,a)—could stem from memorizing attribute labels of specific training images rather than from a general attribute-grounded embedding space. Although the held-out prediction R2 values (0.71–0.96) show the model can predict attributes on unseen patients, the retrieval evidence, which is the concrete demonstration of grounding in embedding space, is compromised. Without a held-out gallery, the conclusion that the embedding space is 'grounded in these important quantitative attributes' (Section 5) is not firmly established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper explores grounding multimodal large language models (MLLMs) in quantitative skin attributes. It fine-tunes Qwen2-VL 7B with LoRA on the SLICE-3D dataset to predict 16 numerical attributes (e.g., lesion area, border jaggedness) from images. The authors propose two embedding functions: an image-only embedding h(im) (average of image-token features) and an attribute-conditioned image-text embedding h(im,a) (final-token feature after appending an attribute-specific question). They evaluate attribute prediction on held-out patients (R2 from 0.71 to 0.96) and present a retrieval case study in which test-set queries retrieve images from the training set, comparing fine-tuned embeddings against vanilla Qwen and MONET baselines. The conclusion claims that the retrieval results provide evidence that MLLM embedding spaces can be grounded in quantitative attributes.","tokens_in":8625,"tokens_out":5177,"duration_ms":59766,"significance":"If the retrieval evidence were sound, the paper would make a useful contribution: it would show that an MLLM's embedding space can be steered by attribute questions, which is relevant for interpretable and interactive dermatology AI. Strengths include the use of a large real-world dataset (SLICE-3D), a held-out patient split for the R2 evaluation, a simple and reproducible fine-tuning setup, and comparison against a medical VLM. However, the central retrieval demonstration is weakened by a training-set gallery and the absence of significance testing. The paper is better viewed as a promising case study than as a fully established demonstration of attribute grounding in embedding space.","major_comments":[{"comment":"The retrieval database is explicitly set to I' = Itr (§3.3), and Itr is exactly the set of images used for LoRA fine-tuning (§4.1). The fine-tuned model therefore had direct supervised access to every gallery image and its attribute values during training, whereas the vanilla Qwen and MONET baselines did not. The consistent ordering in Fig. 4 may reflect training-set familiarity rather than a generally attribute-grounded embedding geometry. This is load-bearing because the conclusion in §5 bases the grounding claim on the retrieval results. Please re-run the retrieval with a held-out gallery (e.g., query from a subset of Ite and retrieve from the remaining Ite, or split Itr into a fine-tuning set and a disjoint gallery), and/or add a baseline that is fine-tuned on Itr without attribute conditioning. If the advantage persists in that setting, the conclusion will be much better supported.","section":"§3.3, §4.1, §4.2 (Fig. 4)"},{"comment":"The narrative claims that the fine-tuned image-only embeddings are 'significantly more effective' than the baselines and that image-text embeddings are 'most effective', but Fig. 4 reports only boxplots of percentile ranks. No confidence intervals, standard errors, or paired significance tests are provided. Since all methods are evaluated on the same query images, paired bootstrap or Wilcoxon signed-rank tests are straightforward. Please add such statistics for at least the key comparisons (fine-tuned image-only vs. MONET, and image-text vs. image-only). Without them, the stated ordering is not quantitatively established.","section":"§4.2, Fig. 4"}],"minor_comments":[{"comment":"The notation h(im,a)(I) = f^{-1}(...) is incorrect as written: f is defined in §3.1 as a feature extractor (a sequence-to-sequence map), not an invertible function. The intended object appears to be the last row of f([g(I), ψ(Q(a))]), i.e., f_{N_I+|Q(a)|}(...). Please fix the definition, as the current form is not reproducible.","section":"§3.3, Eq. (3)"},{"comment":"Please define 'Image Untuned' explicitly in the text or figure caption, and state whether it uses the same image-token averaging as h(im) on the vanilla Qwen model. The current label is ambiguous.","section":"§4.2, Fig. 4"},{"comment":"The R2 computation is described only as 'predict all 16 attributes and compare to ground truth.' For reproducibility, please specify the decoding strategy (e.g., greedy sampling), how the numeric answer is parsed from generated tokens, and whether R2 is computed per value or per attribute after any calibration. Reporting confidence intervals for the R2 values would also be useful.","section":"§4.2, 'Quantitative Prediction'"},{"comment":"The choice of b = 200 is introduced without justification or sensitivity analysis. Since the first-stage image-only embedding h(im) is itself trained with attribute supervision, the hierarchical method is not attribute-neutral; the paper should acknowledge this when interpreting why 'the hierarchical modification does not lower quality.'","section":"§3.3, Hierarchical Retrieval"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a solid, honest proof-of-concept paper. The authors fine-tune Qwen2-VL with LoRA to predict 16 quantitative skin attributes from SLICE-3D images, then use the model's final-token embeddings for attribute-conditioned retrieval. The held-out prediction results are real and reasonably strong: R2 between 0.71 and 0.96 on a test set stratified by patient. That alone is a useful empirical data point, since MLLMs are known to be shaky on numeric properties of images.\n\nWhat's new: the combination of LoRA-tuning an MLLM on quantitative attributes and using question-conditioned final-token embeddings for composed retrieval is a legitimate extension of prior composed-retrieval and MLLM-retrieval work. The idea of using the final token to specialize an embedding toward a specific attribute at inference time is clean and well motivated. The hierarchical retrieval variant is a sensible engineering addition. The writing is clear and the citations look appropriate.\n\nThe soft spot is exactly what the stress-test flags: the retrieval evaluation uses the training set as the image database (Section 3.3 sets I' = Itr, and Itr is the fine-tuning set). The fine-tuned model has seen those images and their attribute labels during training; the vanilla Qwen and MONET baselines have not. So the retrieval advantage, especially for h(im,a), could partly be training-set familiarity, not generalizable embedding geometry. The conclusion that the embedding space is \"grounded in these important quantitative attributes\" rests heavily on this retrieval case study, and the current design doesn't cleanly separate memorization from generalization. That said, the held-out R2 results do provide independent evidence that the model can predict attributes on unseen patients, so the grounding claim isn't empty. The fix is straightforward and the authors can run it: use a held-out gallery (Ite, or a split within Ite) and recompute the boxplots.\n\nMinor issues: no confidence intervals or significance tests on the retrieval percentiles, no code or model release mentioned, and the qualitative retrieval figure is small and subjective. These are minor relative to the gallery issue.\n\nWho this is for: researchers working on interpretable medical AI, dermatology VLMs, or composed retrieval. It deserves a serious referee – it's a clean, honest extension with real empirical content, but the retrieval evaluation needs revision before it can support the stronger grounding claim. I'd be fine citing the R2 result once the gallery issue is addressed. Not a desk reject.\n\nBest,\n[You]","headline":"Honest, useful study: LoRA-tuning Qwen2-VL on 16 SLICE-3D attributes yields held-out R2 0.71–0.96, and the retrieval demo is a genuine extension of composed retrieval, but the retrieval evaluation uses the training set as the gallery, which makes the case for embedding-space grounding weaker than claimed.","tokens_in":9076,"tokens_out":674,"would_cite":true,"duration_ms":9231,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning an MLLM to predict quantitative skin attributes grounds its embeddings in those attributes, and the resulting image-text embeddings retrieve images that match both appearance and the specified attribute.","keywords":["multimodal large language models","quantitative skin attributes","content-based image retrieval","attribute grounding","LoRA fine-tuning","interpretability","SLICE-3D","numerical reasoning"],"falsifier":"Repeat the retrieval experiment with the image database restricted to patient-disjoint held-out images that were never used in LoRA fine-tuning. If the attribute-conditioned embeddings no longer beat the untuned Qwen and medical image-text baselines on attribute match, the reported grounding is an artifact of training-set familiarity rather than a general property of the embedding space.","tokens_in":8242,"feed_emoji":"🔍","tokens_out":8934,"duration_ms":86219,"temperature":0.7,"pith_summary":"The paper claims that fine-tuning a multimodal large language model (MLLM) to predict numerical skin attributes—such as lesion area, border irregularity, and color contrast—can ground the model's embedding space in those attributes. It demonstrates this through a retrieval case study on a dataset of about 400,000 skin lesion images: after LoRA fine-tuning Qwen 2 VL on image-question-answer tuples, the embedding of the final token conditioned on an attribute question retrieves images that match the query both visually and in that attribute. Test-set R² values for predicting the 16 attributes range from 0.71 to 0.96. If the grounding holds, it offers a path toward more interpretable, interactive dermatology AI that can justify its decisions in terms of measurable lesion properties.","feed_headline":"Fine-tuned MLLM embeddings make skin retrieval match numeric traits","feed_subtitle":"Question-conditioned embeddings pull up lesions matching area, color, and border traits, not just appearance.","key_machinery":"The load-bearing object is the attribute-conditioned embedding h(im,a)(I) = f_{-1}([g(I), ψ(Q(a))]), the final decoder token's feature when the input is the image followed by the attribute question. Because the final token attends to every earlier token, this single vector is meant to fuse the whole image with the meaning of the requested attribute. LoRA fine-tuning on numeric answer tokens is what forces that fused representation to carry attribute information. The image-only baseline h(im)(I) averages all image-token outputs and captures general visual similarity, while a hierarchical search first narrows with h(im) and then re-ranks with h(im,a), trading compute for per-attribute precisio","core_discovery":"The paper's central claim is that an MLLM's embedding space can be grounded in quantitative skin attributes by fine-tuning the model to predict those attributes from images. Using Qwen 2 VL 7B, the authors apply LoRA to jointly fine-tune the vision encoder and decoder on 1.4 million (image, attribute question, numeric answer) tuples sampled from the SLICE-3D dataset. They then use the decoder's last-layer final-token embedding, computed from the image plus an attribute-specific question, as an attribute-conditioned image embedding. In retrieval over the training set, this embedding returns top-5 images whose attribute values are closer to the query's than those returned by image-only embeddi","pith_inferences":["Because the retrieval database is the training set itself, part of the fine-tuned model's retrieval advantage could come from having seen those exact images during LoRA training; a held-out retrieval database would cleanly separate attribute grounding from training-set familiarity.","The paper uses a single fixed question per attribute, so retrieval quality may shift with question wording; the reported grounding is for these particular prompts rather than for attribute concepts in general.","High attribute-prediction accuracy does not by itself guarantee that the embedding space is ordered by attribute value; the retrieval percentiles are the direct geometric evidence, and a ranking or linear-probe test on the embeddings would make the geometric claim more explicit.","If the grounding transfers to other image domains with numeric ground-truth attributes, the same question-conditioned final-token recipe could make retrieval systems in radiology or pathology explainable by the quantitative features they match on."],"forward_implications":["Attribute-conditioned retrieval consistently beats image-only retrieval on the 16 tested attributes, so an MLLM can serve as an attribute-aware image search engine after question-answer fine-tuning.","The hierarchical variant, which searches 200 initial images before re-ranking, preserves retrieval quality while avoiding one database per attribute, making the approach practical at SLICE-3D scale.","Because the text prompt can hold several attribute questions at once, multi-attribute retrieval—finding images matching a query on area and border jaggedness simultaneously—is a direct extension of the same embedding recipe.","The reported test-set R² values between 0.71 and 0.96 across all 16 attributes support the claim that the model has learned the numerical concepts on held-out patients."],"supporting_citations":[{"why":"Supplies the SLICE-3D dataset of 401,059 lesion image crops and the 16 quantitative attribute labels used for fine-tuning and retrieval.","marker":"[8]"},{"why":"Establishes that the selected quantitative attributes are predictive of malignancy, motivating their use as interpretable grounding targets.","marker":"[10]"},{"why":"The Qwen 2 VL 7B MLLM whose vision encoder and decoder are fine-tuned with LoRA.","marker":"[16]"},{"why":"The LoRA low-rank adaptation method used to fine-tune the base model on attribute question-answer tuples.","marker":"[5]"},{"why":"A CLIP-like medical image-text foundation model whose image encoder serves as a baseline in the retrieval comparison.","marker":"[6]"},{"why":"Justifies using the final token's decoder output as the embedding because it attends to all earlier image and question tokens.","marker":"[12]"},{"why":"Supplies the composed image-text retrieval idea of merging image and text embeddings, which the image-text embedding function builds on.","marker":"[15]"},{"why":"Documents MLLMs' weakness at numerical properties in images, framing the question of whether attributes can be grounded at all.","marker":"[17]"},{"why":"Provides the precedent of using decoder feature outputs as embeddings for retrieval.","marker":"[21]"}],"fun_headline_variants":["Skin retrieval now matches numeric traits via MLLM grounding","Attribute-grounded MLLM embeddings return closer skin matches","MLLM embeddings tuned to skin attributes in retrieval","Quantitative skin grounding boosts MLLM retrieval","Fine-tuned MLLM aligns embeddings with lesion traits"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The argument assumes retrieval over the training set is a fair test, even though the fine-tuned model was optimized on those exact images while the untuned baselines were not.","fun_headline_variants_meta":{"raw":{"variants":["Skin retrieval now matches numeric traits via MLLM grounding","Attribute-grounded MLLM embeddings return closer skin matches","MLLM embeddings tuned to skin attributes in retrieval","Quantitative skin grounding boosts MLLM retrieval","Fine-tuned MLLM aligns embeddings with lesion traits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1317,"prompt_tokens":711,"completion_tokens":606,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":528}},"tokens_in":455,"tokens_out":606,"duration_ms":6782,"temperature":1.0,"reasoning_tokens":528,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:13:17.980746+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the retrieval experiment with the image database restricted to patient-disjoint held-out images that were never used in LoRA fine-tuning. If the attribute-conditioned embeddings no longer beat the untuned Qwen and medical image-text baselines on attribute match, the reported grounding is an artifact of training-set familiarity rather than a general property of the embedding space.","supporting_citations":[{"cited_title":"The slice-3d dataset: 400,000 skin lesion image crops extracted from 3d tbp for skin cancer detection","cited_arxiv_id":null,"evidence_quote":"Supplies the SLICE-3D dataset of 401,059 lesion image crops and the 16 quantitative attribute labels used for fine-tuning and retrieval."},{"cited_title":"3d whole-body skin imaging for automated melanoma detection","cited_arxiv_id":null,"evidence_quote":"Establishes that the selected quantitative attributes are predictive of malignancy, motivating their use as interpretable grounding targets."},{"cited_title":"Transparent medical image ai via an image–text foundation model grounded in medical literature","cited_arxiv_id":null,"evidence_quote":"A CLIP-like medical image-text foundation model whose image encoder serves as a baseline in the retrieval comparison."},{"cited_title":"Steering llama 2 via contrastive activation addition","cited_arxiv_id":null,"evidence_quote":"Justifies using the final token's decoder output as the embedding because it attends to all earlier image and question tokens."},{"cited_title":"Composing text and image for image retrieval-an empirical odyssey","cited_arxiv_id":null,"evidence_quote":"Supplies the composed image-text retrieval idea of merging image and text embeddings, which the image-text embedding function builds on."},{"cited_title":"Visrag: Vision-based retrieval-augmented generation on multi-modality documents","cited_arxiv_id":null,"evidence_quote":"Provides the precedent of using decoder feature outputs as embeddings for retrieval."}],"review_version":1}