LLaVA-RE: Binary Image-Text Relevancy Evaluation with Multimodal Large Language Model

Jinjin Li; Lan Ma; Oliver Liu; Tao Sun

REVIEW 2 major objections 4 minor 2 references

LLaVA-RE: Binary Image-Text Relevancy Evaluation with Multimodal Large Language Model

T0 review · 2 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Task instructions and in-context examples turn LLaVA into a reliable yes/no relevance judge

desk verdict A useful idea and a clean engineering implementation, but the evaluation is circular: the test labels come from the same heuristic pipeline that made the training data, and one fine-grained protocol actually labels positive pairs as negative. read the letter →

arxiv 2508.05602 v1 pith:I74232CR submitted 2025-08-07 cs.CV

Tao Sun , Oliver Liu , JinJin Li , Lan Ma This is my paper · ORCID

classification cs.CV

keywords binaryrelevancyevaluationimage-textmatchingmultimodallargelanguagemodelsin-contextlearningvisualinstructiontuningLLaVAVQAfine-grainedrelevance

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that binary image-text relevancy evaluation—the yes/no question of whether an image matches a long, multi-format text—can be done well by a multimodal large language model, provided the model is given two things: a detailed task instruction that says what “relevant” means in the current scenario, and a few labeled image-text demonstration pairs drawn from the same scenario. The authors instantiate this as LLaVA-RE on the LLaVA 1.5 architecture and build a new training and evaluation collection from existing public sources, covering multi-turn conversations, Wikipedia paragraphs, recipe ingredient lists, VQA reasoning texts, charts, infographics, and fine-grained category descriptions. They report accuracy between roughly 79.7% and 98.3% across training, hold-out, and unseen tasks, compared with 29.6%–75.4% for the vanilla backbone, and show that in-context demonstration pairs are what pull fine-grained predictions out of a strong “not relevant” bias. If the claim holds, the result is a reusable, instruction-adjustable relevance judge for measuring multimodal response quality and ranking candidate generations.

What carries the argument

The load-bearing mechanism is the prompt template in Eq. (2): a task instruction $S$, then $C$ demonstration pairs $(I_i, T_i)$ with labels $r_i$, then the test pair $(I,T)$. The instruction $S$ disambiguates what counts as relevant, since two otherwise identical pairs can have different labels under different instructions; the in-context samples transfer the label semantics to a new task. The architecture is LLaVA 1.5—a ViT image encoder, an MLP projector, and a Vicuna language model—so the behavior comes from the prompt and instruction-tuning data. The second load-bearing piece is the generated binary relevance dataset: positive pairs are the original data correspondence; negative pairs ar

What would settle it

Have expert annotators relabel a few hundred “hard negative” pairs from the same sources—pairs that share a coarse category but differ in one attribute such as species, color, model year, or the numerical answer in a chart—and run the trained model on them. If accuracy on that curated set drops far below the reported 79.7–98.3 percent range, the measured effectiveness is an artifact of the heuristic labels. A complementary test: on fine-grained tasks, flip the labels of the semantic-related context examples and check whether the model’s predictions follow the flipped labels; if they do, the mo

Watch

Extended reading notes

Core claim

The central discovery is that relevance judgment needs no new architecture: an open MLLM becomes a reliable binary evaluator when the prompt contains (i) a natural-language task instruction fixing the meaning of “relevant” and (ii) a few in-context image-text pairs with labels from the same task. LLaVA-RE implements this on the LLaVA 1.5 architecture with 4-shot multimodal in-context instruction tuning, plus general VQA data to preserve capability. The paper also constructs a binary relevance dataset: positives are original image-text correspondences; negatives are same-category or similarity-based mismatches. It reports accuracies from about 79.7% to 98.3% across training, hold-out, and uns

Load-bearing premise

The load-bearing premise is that the heuristically generated labels—especially the negative pairs made by pairing a text with a similar-but-different same-category image or text—are correct enough that training against them teaches real relevance judgment rather than artifacts of how negatives were sampled.

Editorial extensions

If this is right

Relevance evaluation stops being tied to a fixed embedding model: the same trained weights can judge different definitions of relevance simply by receiving a different task instruction.
A model trained on a few interaction formats transfers to unseen chart and infographic questions with high accuracy, so the approach could be used to screen responses in domains not present in training.
In-context demonstration pairs do real work: at 0 shots, fine-grained predictions collapse toward “not relevant”; adding two to four pairs, especially semantically related ones, restores balance and raises accuracy.
Because general VQA data is mixed into training, the model keeps general visual question-answering ability while acquiring the relevance-judgment skill.
The data-creation pipeline is a viable way to get binary relevance labels at scale from existing public VQA, retrieval, and classification datasets, with no human annotation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not test attribute-level counterfactuals, but the prompt design implies a sharp diagnostic: change one attribute in the text, keep the image fixed, and see whether the model flips its label; aggregate accuracy is too coarse to guarantee this.
Because the paper’s limitation note says heuristic negatives introduce noisy labels, an expert-annotated hard-negative subset would reveal how much of the reported gain survives harder cases; the authors flag this as future work.
The instruction mechanism generalizes beyond relevance: the same prompt-template pattern could build binary evaluators for other judgments, such as safety or preference alignment, by swapping the instruction and the labeled demonstrations. This is an extension, not a claim in the paper.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

I read LLaVA-RE. The core idea is sensible: use an MLLM with task instructions and multimodal in-context examples as a binary image-text relevancy evaluator. That is worth building, and the paper is honest about what it does. The gains over vanilla LLaVA 1.5 are consistent and large across training, hold-out, and unseen tasks, which suggests the instruction/ICL design is doing something. The dataset, while unreleased, covers a useful spread of task formats.

But the central effectiveness claim is not supported, and the stress-test note lands. Test labels are generated by the same heuristic pipeline that created the training labels: same-category sampling, similarity thresholds, Claude-written instructions. The paper itself admits the labels are noisy. That alone would be a serious problem, because accuracy on self-labeled data can reflect learning the heuristic, not learning relevancy. The fine-grained protocol is worse: the positive pair is an image and its class description, but the negative pair is another image from that same fine-grained class. Since the description describes the class, that other image is relevant by the paper's own instructions. Those are positive pairs labeled negative, and they are in both train and test. The reported 80-90% accuracies on fine-grained tasks are therefore not measuring relevancy in any meaningful sense.

There are softer problems too: no human-validated ground truth, no error bars, test sets as small as 300-1,000 samples, and only vanilla LLaVA as baseline. The paper compares to no other MLLM and no retrieval-based method. Code and data are not released. All of this is fixable, but as submitted the experiments do not validate the framework.

I still think this deserves refereeing rather than desk rejection. The task is relevant, the formulation is clear, and the authors are transparent about limitations. A competent reviewer could push for a human-annotated test set and stronger baselines. The paper is a reasonable starting point, but it is not yet a demonstrated result. I would not cite the effectiveness numbers, though I'd keep it in mind as related work if a revised version appears.

Referee Report

2 major / 4 minor

Summary. The paper proposes LLaVA-RE, a LLaVA 1.5 model fine-tuned for binary image-text relevancy evaluation. The method augments the standard LLaVA prompt with detailed task instructions and multimodal in-context learning samples. Since no public dataset exists for this task, the authors construct a new dataset from several public sources (LLaVA-Instruct, WIT, RecipeQA, TextVQA, TDIUC, ChartQA, InfographicVQA, and fine-grained classification sets), using automated heuristics to create positive and negative image-text pairs. The paper reports accuracy improvements over vanilla LLaVA 1.5 on training tasks, held-out tasks, unseen tasks, and six fine-grained tasks, and concludes that the framework is effective.

Significance. If the evaluation were valid, LLaVA-RE would be a useful practical contribution: binary relevancy evaluation is important for generative-AI response ranking, and the design of combining task instructions with ICL is reasonable. The paper also addresses a real gap, as most image-text matching models are evaluated on short captions rather than complex texts. The authors are candid about some limitations, and the architecture choices are well motivated. However, the central claim rests entirely on accuracy numbers computed against automatically generated labels. Because those labels are produced by the same heuristics used to create training data, and because the fine-grained negative construction is logically inconsistent with the task definition, the reported results do not currently establish that LLaVA-RE understands image-text relevancy. The contribution is therefore promising in conception but unsupported by the present evidence.

major comments (2)

[Appendix A.1 (Fine-grained tasks), Fig. 5] The fine-grained negative-pair construction is internally inconsistent. For each class, Claude 3 Sonnet generates a textual description of the class, the positive image is an instance of that class, and the negative image is 'randomly sample another image from the same fine-grained class.' By the paper's own instruction to 'say yes if the image shows exactly the same fine-grained attributes as the description,' any image of the same fine-grained class satisfies the description. These negatives are therefore positive pairs labeled as negative. The fine-grained accuracy and ICL conclusions in Fig. 5 are built on this invalid ground truth, so they cannot support the claimed effectiveness of ICL for relevance evaluation.
[Sec. 3.3, Appendix A.1, Table A.1; Limitations] The evaluation signal is circular. All test labels are generated by the same automatic heuristic rules used to produce training labels: LLaVA negatives sample another image from the same COCO category, WIT negatives pair page descriptions with section images, RecipeQA negatives come from the same recipe's negative food images, and TextVQA/TDIUC negatives are sampled using image or text similarity. A model can therefore achieve high accuracy by imitating the sampling heuristic rather than by judging true relevance. Because no human-validated labels are provided, the large gains over vanilla LLaVA in Table A.1 (e.g., wiki from 49.3 to 97.7 for 7B) cannot be interpreted as evidence of effectiveness. The Limitations paragraph on 'Label noises' concedes that the procedure 'inevitably introduces some noisy labels' and that similarity scores 'may not necessarily reflect the true fine-grained co

minor comments (4)

[Throughout] There are several typos: 'imaga-text pairs' (Related Work), 'hae' (Related Work), 'tokenizor' (Sec. 3.4), 'sematic-related' (Fig. 5 caption), 'Langauge' (Introduction). Please proofread.
[Sec. 4.2, Figs. 3-5] No confidence intervals, standard deviations, or significance tests are reported. Some test sets are very small (wiki and tdiuc have 300 samples; recipe, chartqa, and infographics have 1,000), so differences of a few percent between configurations may not be stable. Reporting variance or at least sample counts in the figures would help.
[Sec. 4.2, Table A.1] The comparison is only against vanilla LLaVA 1.5. No other MLLMs (e.g., GPT-4V, Qwen-VL, InternVL) or traditional retrieval models are evaluated. This limits the scope of the 'effectiveness' claim even if the label-validity issue were resolved.
[Sec. 3.1 vs. A.3] The formulation in Eq. (1) defines the output label as 'Relevant' or 'Not Relevant,' but several task-instruction examples in Appendix A.3 instruct the model to answer with a single word 'Yes' or 'No.' This inconsistency should be clarified.

Circularity Check

2 steps flagged · score 6.0 of 10

Test labels are generated by the same heuristic pipeline used for training, and the fine-grained negative pairs are positive by the paper's own task-instruction definition; reported accuracies partly measure reproduction of the labeling rule.

self definitional [Sec 3.3 (data creation) + Sec A.1 (per-task negative sampling) + Sec 4.1 (evaluation)]
"Positive pairs are easy to obtain as they can be derived from the raw data correspondences, while negative pairs may not exist. We create negative pairs by sampling images (or texts) from the same category."

The model is trained on labels produced by this heuristic and then evaluated on test labels produced by the same heuristic: Appendix A.1 reuses the same rule for every task (e.g., for TextVQA 'we randomly sampled another image from the same category based on image similarity scores'; ChartQA and Infographics are created 'in a similar manner'). The reported accuracy is therefore agreement with the annotation rule that defines the labels, not relevancy against an independent ground truth. The central claim that 'experimental results validate the effectiveness of our framework' reduces to reproducing the construction. The Limitations section concedes the signal is not clean: 'this procedure inevitably introduces some noisy labels.'
self definitional [Appendix A.1, Fine-grained tasks; cf. Sec A.3 Cars task instruction]
"The positive pair is an image and the corresponding class description. To get a negative pair, we randomly sample another image from the same fine-grained class."

Under the paper's own task-instruction definition for these tasks (e.g., Cars: 'Only say yes if the image shows exactly the same fine-grained attributes as the description'), any image from the same fine-grained class matches the class description and is therefore relevant. Sampling another image from the same fine-grained class as a negative labels a positive pair as negative by the paper's own definition. Because the same construction feeds both training and test splits, accuracy on the fine-grained tasks measures consistency with an internally contradictory labeling rule, not image-text relevancy.

full rationale

Score 6. The central empirical claim is substantially circular in its evaluation: LLaVA-RE is trained on a binary-relevancy dataset whose negative pairs are produced by heuristic sampling rules, and the test splits for training, hold-out, and unseen tasks are produced by the same rules (Sec A.1). The model's 'effectiveness' is therefore partly self-defined by the label-construction procedure; the Limitations section explicitly concedes that the heuristic 'inevitably introduces some noisy labels' and that image similarity 'may not necessarily reflect the true fine-grained correlations.' The fine-grained tasks are worse: a same-fine-grained-class image is treated as negative even though the class description describes that class, so the label is contradictory under the paper's own instruction. This is a data/evaluation circularity, not a citation-chain circularity: LLaVA-RE builds on external work (LLaVA 1.5, Doveh et al.) and there are no load-bearing self-citations. The score is 6 rather than lower because the method itself is a genuine architecture/training contribution and the comparison to vanilla LLaVA still shows the model learns something about the constructed labels; however, until the test labels are validated by independent human annotation or another external signal, the accuracy numbers do not establish true relevancy evaluation. Score is not 8-10 because the paper does not fit a parameter to the test set and the construction is an acknowledged noisy-label issue rather than a theorem whose conclusion is identical to its premise.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The paper's conclusions depend on a small set of unvalidated assumptions about data labeling and transfer, which are not independently verified.

free parameters (4)

Negative-pair similarity threshold = unspecified
Negative images (or texts) are sampled based on image or text similarity scores in several tasks (Sec A.1); the metric and threshold are not given, yet they control task difficulty and the resulting labels.
Number of ICL context samples = 4 (training), 0/2 (inference)
Chosen mainly due to the 4096-token limit; the paper notes 2-shot inference shows little gain and semantic contexts outperform random, so the design is not optimized.
LoRA rank and alpha = not stated
Instruction tuning uses LoRA on 8x A100s, but rank and alpha are not reported; these hyperparameters affect capacity and results.
Train/test split sizes = e.g., llava 10k/6k, wiki 20k/300, recipe 12k/1k
Split ratios are chosen ad hoc without justification; small test sets like 300 samples yield high-variance accuracy estimates.

assumptions (5)

ad hoc to paper The automatic labeling pipeline creates valid ground-truth relevancy labels.
The paper's train and test labels are generated by heuristics such as sampling same-category images or using similarity scores. The authors acknowledge in Limitations that similarity may not reflect true correlations and that the procedure introduces noisy labels. The entire evaluation rests on this assumption being approximately true.
domain assumption Positive pairs derived from raw dataset correspondences (e.g., an image and its own QA) are genuinely relevant.
Used in Sec A.1 for all tasks; mostly reasonable for VQA, recipe, and Wikipedia entries, but the definition of relevancy is task-dependent and unvalidated.
domain assumption Claude 3 Sonnet-generated justifications and fine-grained class descriptions are accurate and relevant.
Claude is used to expand texts and to create fine-grained class descriptions (Sec A.1); no verification of accuracy is provided, and errors would propagate into both training and test.
domain assumption LLaVA 1.5 pretrained weights provide a reasonable base for relevancy evaluation.
Standard transfer-learning assumption; LLaVA 1.5 is cited and its weights are used without further justification.
domain assumption A task instruction S is sufficient to disambiguate the meaning of relevancy.
Formulation in Eq. (1) assumes an instruction clarifies relevancy; Sec 3.1 states this assumption, and the Limitations note that instruction-following depends on the backbone model.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LLaVA-RE: Binary Image-Text Relevancy Evaluation with Multimodal Large Language Model." pith.science (2026). https://pith.science/paper/I74232CR

@misc{pith2026250805602,
  author       = {Pith},
  title        = {Pith review of: LLaVA-RE: Binary Image-Text Relevancy Evaluation with Multimodal Large Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I74232CR}},
  note         = {Machine review of arXiv:2508.05602}
}

read the original abstract

Multimodal generative AI usually involves generating image or text responses given inputs in another modality. The evaluation of image-text relevancy is essential for measuring response quality or ranking candidate responses. In particular, binary relevancy evaluation, i.e., ``Relevant'' vs. ``Not Relevant'', is a fundamental problem. However, this is a challenging task considering that texts have diverse formats and the definition of relevancy varies in different scenarios. We find that Multimodal Large Language Models (MLLMs) are an ideal choice to build such evaluators, as they can flexibly handle complex text formats and take in additional task information. In this paper, we present LLaVA-RE, a first attempt for binary image-text relevancy evaluation with MLLM. It follows the LLaVA architecture and adopts detailed task instructions and multimodal in-context samples. In addition, we propose a novel binary relevancy data set that covers various tasks. Experimental results validate the effectiveness of our framework.

Figures

Figures reproduced from arXiv: 2508.05602 by the authors.

**Figure 2.** Data creation pipeline. from training data of the same relevancy task. The task instruction together with demonstration examples form the prompt input for MLLMs. It can vary across different samples. 3.3 Binary Relevancy Data Creation As there are no available complex binary relevancy datasets for training and evaluation, we create data from diverse public datasets listed in Tab. 1. These are for preliminary experi… view at source ↗

**Figure 3.** Evaluation results on training and unseen tasks. ( [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Effect of task instructions on LLaVA 1.5-7B. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 1 canonical work pages

[2014]

Question: {Q} Answer: {A}

Food-101–mining discriminative components with random forests. In European Conference on Computer vision, pages 446–461. Springer. Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. 2024. Matryoshka multimodal models. arXiv preprint arXiv:2405.17430. Min Cao, Shiping Li, Juntao Li, Liqiang Nie, and Min Zhang. 2022. Image-text retrieval: A survey on re-...

arXiv 2024
[2022]

Dodge Caliber Wagon 2007

comprises a diverse collection of infograph- ics with question-answer annotations. The ques- tions require elementary reasoning and basic arith- metic skills over the document layout, textual con- tent, graphical elements and data visualization. We use Claude 3 Sonnet to generate a few sentences to justify the answer, and create positive/negative pairs in...

work page 2013

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [2014]

Question: {Q} Answer: {A}

Food-101–mining discriminative components with random forests. In European Conference on Computer vision, pages 446–461. Springer. Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. 2024. Matryoshka multimodal models. arXiv preprint arXiv:2405.17430. Min Cao, Shiping Li, Juntao Li, Liqiang Nie, and Min Zhang. 2022. Image-text retrieval: A survey on re-...

arXiv 2024

[2] [2022]

Dodge Caliber Wagon 2007

comprises a diverse collection of infograph- ics with question-answer annotations. The ques- tions require elementary reasoning and basic arith- metic skills over the document layout, textual con- tent, graphical elements and data visualization. We use Claude 3 Sonnet to generate a few sentences to justify the answer, and create positive/negative pairs in...

work page 2013