{"id":"c8c73410-1097-4878-9318-57ee6a93d018","arxiv_id":"2411.18038","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"VLM-HOI distills BLIP image-text matching scores into an HOI detector via a contrastive loss, achieving 34.25 mAP on HICO-DET and 67.7 AP on V-COCO with a ResNet-50 backbone.","lead":"This paper presents VLM-HOI, a training-time method that uses a frozen vision-language model (BLIP) to score how well candidate human-object-interaction triplets match the image, and distills those scores into a transformer-based HOI detector with a contrastive loss. The method reports top ResNet-50 results on HICO-DET and V-COCO, but the gains over a CLIP-based predecessor are modest and the paper contains several internal inconsistencies.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The VLM objective is underspecified and internally contradictory: Eq. (2) makes positive and negative sentences identical, and Eq. (4)'s direction reverses the prose, so the reported gains cannot be attributed to BLIP supervision without a corrected specification and a discriminability check.","rationale":"The reader's verdict is CONDITIONAL, and my analysis supports that condition rather than moving to a stronger verdict. The most load-bearing issue is that the paper's central mechanism cannot be evaluated as written: Eq. (2) does not specify a negative sentence, and the prose in Sec. 3.4 reverses the optimization direction of Eq. (4). These are not cosmetic errors; they determine the gradient signal, and a reader implementing the literal text would not obtain the reported method. The reader's weakest_assumption concerns BLIP score reliability, which is real but secondary: even before asking whether BLIP scores are discriminative, the paper must say what the scores are being compared against and in which direction. My proposed BLIP discriminability test would settle the empirical half of the concern; if it fails, the central claim would be not just unverifiable but unsupported. Since the paper could in principle be fixed by supplying code, a corrected loss, and a concrete Sneg construction, the appropriate verdict remains CONDITIONAL rather than REJECT or UNVERDICTED. I therefore leave the verdict unchanged, with the condition that the authors resolve the internal contradictions and provide a reproducible specification.","tokens_in":12235,"tokens_out":8426,"duration_ms":78338,"concrete_test":"Compute BLIP ITM scores for a sample of 2,000 images from HICO-DET with their ground-truth HOI triplets rendered as 'A person {verb} a {object}', and for matched negative triplets formed by randomly permuting the verb across the same object set. Measure the separation (AUC, or mean positive minus mean negative score). If the AUC is below about 0.7 or the ordering is inverted relative to Eq. (4), the proposed objective does not provide reliable supervision and the reported gains are unlikely to replicate. This check uses only the released BLIP checkpoint and the public dataset, so it is independent of the paper's implementation and would tell us whether the VLM signal is real.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a frozen BLIP image-text matching score, applied to template sentences derived from predicted HOI triplets, provides a training signal that improves HOI detection beyond CLIP-based distillation. For this to hold, two things must be true: (i) BLIP ITM scores must be discriminative for the exact prompt format used, and (ii) the loss described must be the loss that was actually optimized. The paper fails to establish (i) and contradicts (ii). In Sec. 3.3, Eq. (2) defines the positive sentence s+_i and negative sentence s-_i with identical text ('A person {verb} a {object}'); no construction of Sneg is given anywhere. In Sec. 3.4, the prose says positive scores should be 'as close to zero as possible' and negative scores 'as high as possible,' whereas Eq. (4) pushes sim(I, s+) above margin alpha and minimizes sim(I, s-). Table 1 and the text also disagree on the default mAP (34.25 vs 33.64). As written, the training objective is not a well-defined function of the predictions, and there is no way to determine whether the reported gains arise from the VLM objective, from other loss terms, or from uncontrolled differences in training. The only evidence for BLIP discriminability is Fig. 1, an anecdotal comparison on a few triplets. This is the load-bearing soft spot: the method's mechanism is unverified and its specification is self-contradictory.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VLM-HOI, a method that uses a frozen BLIP image-text matching (ITM) model as an external objective for training a DETR-based HOI detector. Predicted HOI triplets are converted into template sentences, and BLIP's ITM scores for positive and negative sentences are combined into a contrastive loss. The authors report state-of-the-art results on HICO-DET and V-COCO. However, as written, the objective is underspecified and internally contradictory: Eq. (2) defines positive and negative sentences identically, Eq. (4) contradicts its own prose, and reported numbers disagree between text and tables. These issues prevent the reader from reproducing or verifying the claimed contribution.","tokens_in":12460,"tokens_out":4972,"duration_ms":41868,"significance":"The underlying idea--using a frozen, object-centric VLM as a training objective that is not needed at inference time--is of interest to the HOI community and is a natural extension of CLIP-based distillation methods. The qualitative evidence in Fig. 1 that BLIP ITM is more discriminative than CLIP for short, ungrammatical triplets is suggestive. If the objective were properly specified and the numbers reconciled, the method could be a useful baseline for future work. The current manuscript, however, does not yet provide a well-defined training loss, and the reported gains cannot be attributed to the proposed mechanism. No code is released, which further limits reproducibility.","major_comments":[{"comment":"The definition of positive and negative sentences is identical: s+_i and s-_i are both [\"A person\" + v_i + \" a \" + o_i], differing only in the condition T in H vs T not in H. Since no actual construction of Sneg is given--e.g., which verbs/objects are swapped, whether negatives are sampled from the dataset, or whether they are the same template with random labels--the loss in Eq. (4) is not a well-defined function of the model predictions. This must be specified before the reported gains can be attributed to the proposed objective.","section":"Section 3.3, Eq. (2)"},{"comment":"The text states that positive similarity scores should be \"as close to zero as possible\" and negative scores \"as high as possible,\" which is the opposite of Eq. (4), where positive scores are pushed above margin alpha and negative scores are minimized. The direction of the objective must be corrected and made consistent with Figure 1, which shows positive scores as high. As written, the training objective is ambiguous and cannot be used to interpret the experimental results.","section":"Section 3.4, Eq. (4) and surrounding text"},{"comment":"The text reports 33.64% mAP on HICO-DET Default, while Table 1 reports 34.25; similarly, the V-COCO margins over GEN-VLKT are stated as 7.1 and 7.7 AP, whereas Table 2 implies 5.3 and 6.5 AP. These numeric inconsistencies affect the central claim of improvement over GEN-VLKT and must be reconciled before the results can be verified.","section":"Section 4.4, Table 1 and text"},{"comment":"The state-of-the-art claim is based on an incomplete baseline set. The tables omit RLIPv2 and HOICLIP, both of which are closely related language-supervised HOI methods cited in the text, and the exclusion of Swin-based models is stated only in the table caption while the abstract and title make an unqualified SOTA claim. The comparison should be completed or the claim narrowed to match the actual baselines.","section":"Section 4.4, Tables 1-2"},{"comment":"The positive margin alpha is selected using V-COCO test performance, with no validation set described; the reported AP values in Table 3 are test-set numbers. This selection-on-test procedure can inflate the reported gains and should be clarified or replaced by validation-based hyperparameter selection, especially since alpha is a key component of the proposed loss.","section":"Section 4.5, Table 3"}],"minor_comments":[{"comment":"The heading \"Vision-Lauguage Model\" contains a typo; it should read \"Vision-Language Model.\"","section":"Section 2.2 heading"},{"comment":"The sentence \"Another content [16,18,26,34,47,52] recent work has focused on improving both branches of HOI detection\" is ungrammatical and should be rephrased.","section":"Section 2.1"},{"comment":"The notation is inconsistent: the text defines object and interaction class names as o_i and v_i, but Eq. (2) uses hat v_i and hat o_i; unify the notation.","section":"Section 3.3, Eq. (2)"},{"comment":"Figures 1 and 5 lack axis labels and score scales, so the claimed typical positive-score range of 1 to 2 in Section 4.5 cannot be verified.","section":"Figures 1 and 5"},{"comment":"The claim of \"first utilization of VLM language abilities for HOI detection\" is too strong given the cited GEN-VLKT and RLIP works; the claim should be qualified, e.g., \"first to use a frozen image-text matching VLM as a training objective for HOI detection.\"","section":"Abstract"},{"comment":"The parameter counts are confusing: the table lists 430M (DETR+BLIP) and 28M (Learnable), while the text says BLIP has 361M and the baseline has 41M DETR plus 28M decoder; clarify the composition of these numbers.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads like an early preprint with substantial presentation and specification problems. The editor should require a corrected specification of the loss, consistent numbers across text and tables, and a more complete comparison before further consideration. The hyperparameter selection on the test split in Section 4.5 should also be addressed. The core idea is publishable if these issues are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The actual new thing is narrow but real: instead of distilling CLIP embeddings into an HOI detector like GEN-VLKT and HOICLIP, the authors freeze BLIP and use its image-text matching score on template sentences as a contrastive training signal. That is a legitimate extension of the existing program, and it has a genuine practical virtue: no inference-time cost, only about 2.4x training time. On HICO-DET the reported gain over GEN-VLKT is small (about 0.5 mAP default in the table), and on V-COCO it is larger (5-6 AP in the table, 7-8 in the text). The prompt ablation is the right kind of experiment.\n\nThe soft spots are not minor. Eq. (2) literally defines the positive and negative sentences by the same template, and no construction of Sneg is given anywhere. Sec. 3.4's prose says positive scores should be close to zero and negative scores high, which reverses Eq. (4), the actual contrastive loss. Table 1 says VLM-HOI gets 34.25 default mAP while the text says 33.64; the V-COCO numbers differ by 1.8-2.2 AP between text (69.5/72.1) and table (67.7/70.9). HOICLIP is cited but omitted from the comparison, and the abstract's 'first utilization of VLM language abilities for HOI detection' is not right on the paper's own showing, since GEN-VLKT and HOICLIP are CLIP-based methods. The margin alpha is tuned directly on the evaluation split. No code is released.\n\nI agree with the stress-test: the central mechanism is unverified. The only evidence that BLIP's ITM scores are discriminative on malformed prompts like 'A person hold a tennis racket' is Fig. 1, a few anecdotal triplets. None of this proves the reported numbers are wrong; it means the paper as written cannot be checked, and the gains cannot be attributed to the BLIP objective rather than to other loss terms or training differences.\n\nWho is this for? Researchers working on VLM distillation for structured prediction. The idea is worth discussing and potentially citing once fixed. As submitted, I would not rely on it. But it deserves a serious referee: a clear revision with consistent equations, a specified negative-sentence sampler, HOICLIP in the table, a discriminability analysis of ITM scores, and code would turn this into a testable, useful paper. Send it to review, but expect major revision.","headline":"A plausible VLM-distillation extension with a real practical upside, but the central loss is under-specified and the reported numbers don't line up; fixable, not trustworthy as written.","tokens_in":13137,"tokens_out":3688,"would_cite":false,"duration_ms":33629,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen vision-language model can serve as a training objective for HOI detection, beating CLIP-based knowledge transfer.","keywords":["vision-language model","human-object interaction detection","knowledge distillation","contrastive learning","image-text matching","BLIP","HOI detection","DETR"],"falsifier":"One concrete check would be to evaluate the frozen BLIP teacher alone on the HICO-DET test set: compute ITM scores for the ground-truth triplets (positive sentences) and for the same triplets with the verb randomly replaced (negative sentences), and measure the separation between the two score distributions. If the distributions overlap heavily, the contrastive loss cannot carry meaningful supervision, and the reported gains would not replicate. A second direct test is to replace the BLIP teacher with a randomly initialized BLIP of the same architecture; if VLM-HOI retains its advantage, the gains come from the extra loss term rather than from language priors.","tokens_in":11938,"feed_emoji":"🖼️","tokens_out":6769,"duration_ms":51374,"temperature":0.7,"pith_summary":"The paper tries to establish that a frozen vision-language model can serve as the objective function for human-object interaction (HOI) detection, not just as a feature extractor. The proposed VLM-HOI pipeline converts each predicted HOI triplet into a template sentence like \"A person hold a tennis racket\", scores it against the image with BLIP's image-text matching, and uses those scores in a contrastive loss. The authors report state-of-the-art mAP on HICO-DET and V-COCO, surpassing GEN-VLKT, a CLIP-based knowledge-transfer method. If the claim holds, it shows that object-centric VLMs encode priors that can regularize and improve HOI prediction, especially for rare categories, without adding any computation at inference time.","feed_headline":"Frozen BLIP as loss function beats CLIP teachers in HOI detection","feed_subtitle":"A contrastive image-text score supervises HOI training; no extra model is needed at inference.","key_machinery":"The key machinery is the image-text matching (ITM) score $\\mathrm{sim}(I, S)$ of a frozen BLIP model, applied to template sentences \"A person [verb] a [object]\" constructed from predicted HOI triplets. These scores feed a contrastive loss $\\mathcal{L}_{\\mathrm{ITM}} = \\sum_i \\max(0, \\alpha - \\mathrm{sim}(I, s^+_i)) + \\sum_j \\mathrm{sim}(I, s^-_j)$, which regularizes a DETR-based HOI detector during training only. BLIP is chosen over CLIP because its object-centric, sub-word-level matching is claimed to separate positive from negative triplet sentences more sharply, as illustrated in the paper's comparison figures.","core_discovery":"On the paper's terms, the discovery is that BLIP's image-text matching score, computed for short grounded template sentences describing predicted human-object-verb triplets, is a usable training signal for HOI detection. The score is turned into a contrastive loss with a positive margin that pulls matched triplets toward the image and pushes unmatched triplets away, and this loss is added to the standard HOI detection loss while the VLM stays frozen. Experiments on HICO-DET and V-COCO show improvements over prior methods, including the CLIP-based GEN-VLKT, with the largest gains on rare interaction categories. The authors argue this is the first use of a VLM's language abilities as an objective function for HOI detection.","pith_inferences":["Beyond the paper, the same training-time objective could be applied to video HOI detection, using a video-language model's matching scores across frames, if such scores are discriminative enough.","Because the paper never specifies how negative sentences are built, an immediate test is whether hard negatives (verbs swapped among objects present in the image) give larger gains than random negatives; the contrastive loss's behavior will hinge on that choice.","A natural extension would be to use a larger, instruction-tuned VLM as the teacher; the paper's own argument implies gains should scale with the teacher's language understanding, which is testable by replacing BLIP with a larger VLM.","If BLIP's scores are sensitive to the awkward sentence template, then dataset-specific verb and object vocabulary changes could break the transfer, making prompt engineering part of the method's deployment cost."],"forward_implications":["A frozen VLM can act as a training-time teacher for HOI detection, so the deployed model has the same parameter count and inference speed as the baseline.","Object-centric VLMs such as BLIP may be better-suited teachers than CLIP for tasks that require localizing subjects and objects, not just global image-text alignment.","Rare HOI categories stand to gain the most from the VLM's language priors, since the contrastive signal regularizes categories with few training examples.","The template-based ITM objective could extend to other structured prediction tasks that can be expressed as (subject, verb, object) text.","The choice of the positive margin $\\alpha$ is tied to the scale of the teacher's scores, so a principled setting of $\\alpha$ depends on the score distribution of the chosen VLM."],"supporting_citations":[{"why":"Supplies the frozen BLIP VLM whose image-text matching scores carry the distillation signal.","marker":"[21]"},{"why":"The CLIP-based knowledge-transfer method GEN-VLKT that VLM-HOI is compared against and outperforms.","marker":"[26]"},{"why":"The same-backbone DETR-based HOI baseline whose architecture VLM-HOI builds on.","marker":"[18]"},{"why":"Provides the alternative CLIP teacher that the paper argues is inferior for localization.","marker":"[35]"},{"why":"Provides the DETR initialization for the object detector.","marker":"[5]"},{"why":"Provides the HICO-DET benchmark and its 600 HOI categories used for evaluation.","marker":"[14]"},{"why":"Provides the V-COCO benchmark and evaluation scenarios used for evaluation.","marker":"[13]"}],"fun_headline_variants":["VLM scores supervise HOI detection beating CLIP baselines","BLIP's matching score trains HOI model without extra inference cost","First to use VLM as loss for HOI: BLIP beats CLIP","Interpretable HOI via VLM objective: SOTA on HICO-DET and V-COCO"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that BLIP's image-text matching scores on short, grammatically awkward template sentences are reliable and discriminative enough to separate correct from incorrect HOI triplets, even though the sentences are unlike BLIP's natural-caption training data and the construction of negative sentences is left unspecified.","fun_headline_variants_meta":{"raw":{"variants":["VLM scores supervise HOI detection beating CLIP baselines","BLIP's matching score trains HOI model without extra inference cost","First to use VLM as loss for HOI: BLIP beats CLIP","Interpretable HOI via VLM objective: SOTA on HICO-DET and V-COCO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000577,"raw_usage":{"total_tokens":2693,"prompt_tokens":888,"completion_tokens":1805,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":1721}},"tokens_in":504,"tokens_out":1805,"duration_ms":12410,"temperature":1.0,"reasoning_tokens":1721,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:34:31.950303+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One concrete check would be to evaluate the frozen BLIP teacher alone on the HICO-DET test set: compute ITM scores for the ground-truth triplets (positive sentences) and for the same triplets with the verb randomly replaced (negative sentences), and measure the separation between the two score distributions. If the distributions overlap heavily, the contrastive loss cannot carry meaningful supervision, and the reported gains would not replicate. A second direct test is to replace the BLIP teacher with a randomly initialized BLIP of the same architecture; if VLM-HOI retains its advantage, the gains come from the extra loss term rather than from language priors.","supporting_citations":[{"cited_title":"In: International Con- ference on Machine Learning","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen BLIP VLM whose image-text matching scores carry the distillation signal."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"The CLIP-based knowledge-transfer method GEN-VLKT that VLM-HOI is compared against and outperforms."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"The same-backbone DETR-based HOI baseline whose architecture VLM-HOI builds on."},{"cited_title":"In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16","cited_arxiv_id":null,"evidence_quote":"Provides the HICO-DET benchmark and its 600 HOI categories used for evaluation."}],"review_version":1}