OVFact: Measuring and Improving Open-Vocabulary Factuality for Long Caption Models

Anurag Arnab; Cordelia Schmid; Monika Wysocza\'nska; Shyamal Buch

REVIEW 3 major objections 5 minor 19 references

OVFact: Measuring and Improving Open-Vocabulary Factuality for Long Caption Models

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a reference-free, open-vocabulary factuality score for long captions, built from visual grounding and an F1 of precision and recall, can both measure and, through data filtering, improve vision-language model…

desk verdict OVFact is a genuine step forward for long-caption factuality metrics, but the headline filtering gains are partly self-measured and the recall definition is the soft spot. read the letter →

arxiv 2507.19262 v1 pith:7OZD5FUC submitted 2025-07-25 cs.CV

Monika Wysoczańska , Shyamal Buch , Anurag Arnab , Cordelia Schmid This is my paper · ORCID

classification cs.CV

keywords open-vocabularyfactualitylongcaptiongenerationobjecthallucinationreference-freeevaluationdatafilteringvisualgroundingprecision-recallmetricvision-languagemodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tackles a measurement problem: long captions produced by vision-language models (VLMs) often describe objects that are not actually in the image, but existing hallucination metrics are tied to short captions and to human-annotated object labels. It introduces OVFact, a score that parses a caption into candidate entities, grounds each entity in the image with open-vocabulary detection and segmentation tools, and combines precision (the fraction of mentioned objects that are real) with recall (the fraction of an image's key objects that are mentioned) into an F1 score. Because the score needs no ground-truth captions, it can also be used to filter large noisy training sets. The paper reports that models trained on an OVFact-filtered subset, 2.5 to 5 times smaller than the full noisy set, improve factuality precision without losing descriptiveness on downstream long-caption benchmarks, and that the metric agrees with human judgments better than prior metrics.

What carries the argument

The load-bearing object is the $\mathrm{OVFact}_{\mathrm{F1}}$ score: $\mathrm{OVFact}_{\mathrm{Prec}} = |G|/|C|$, the ratio of grounded to candidate entities, and $\mathrm{OVFact}_{\mathrm{Rec}}$, the average over reference entities of their maximum cosine similarity to candidate entities, combined as a harmonic F1. The reference set comes either from ground-truth captions when available or, in the fully reference-free setting, from grounding tools run over a 2792-concept open-vocabulary vocabulary built from standard visual datasets. The F1 form matters because it rewards captions that both avoid hallucinated objects and retain descriptive coverage, and it is the exact scoring function used to rank and prune training pairs in the data-filtering experiments.

What would settle it

One could test the claim by taking long captions with known human-annotated objects, deliberately selecting objects outside the union of Visual Genome, LVIS, Open Images, and Objects365, and checking whether OVFactRec and the filtering ranking penalize captions that mention only those out-of-vocabulary objects; a systematic drop in agreement with human judgments on that subset would show the vocabulary bounds the metric.

Watch

Extended reading notes

Core claim

The central claim is that object-level factuality of long, free-form captions can be measured reliably without human-annotated references by combining an LLM-based entity parser, open-vocabulary object detection and segmentation for grounding, and a text-embedding maximum-similarity recall against a 2792-concept vocabulary. This produces two interpretable scores, $\mathrm{OVFact}_{\mathrm{Prec}}$ and $\mathrm{OVFact}_{\mathrm{Rec}}$, unified in $\mathrm{OVFact}_{\mathrm{F1}}$. The reference-free property is load-bearing: it is what allows the metric to score and prune VLM-generated pretraining data, where no ground truth exists. The empirical claim is that filtering a large noisy caption set by $\mathrm{OVFact}_{\mathrm{F1}}$ and fine-tuning a VLM on the top 20\%--40\% of samples yields captions with higher factual precision, equal or better recall, lower hallucination scores on the traditional CHAIR benchmark, and human preference in 68.2\% of side-by-side comparisons against a model trained on the full dataset.

Load-bearing premise

The metric assumes that grounding a fixed 2792-concept vocabulary with open-vocabulary detection and segmentation, matched by maximum cosine similarity, reliably captures every object a good long caption should mention; if important visual concepts fall outside that vocabulary or the tools miss them, both the recall score and the filtering decisions are biased.

Editorial extensions

If this is right

OVFact can be applied to any image-caption pair without human annotations, extending factuality measurement to newly generated datasets and deployment settings where ground-truth captions do not exist.
Filtering by OVFact yields better factuality at smaller training sizes, so training long-caption models can become substantially more data-efficient while improving output quality.
Because OVFact includes recall, it removes the incentive for a captioning model to game precision by producing short, conservative captions that mention few objects.
The same filtering signal improves standard hallucination metrics such as CHAIR, showing the method transfers beyond long-caption benchmarks.
The improvements appear across model scales, suggesting the filtering benefit is not specific to one architecture size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the reference-free design holds up, the same machinery could be turned into a real-time factuality monitor for deployed captioning systems, since the grounding tools use late-fusion architectures where adding thousands of text queries costs only a few percent extra runtime.
The recall side depends on the 2792-concept vocabulary matching what a good caption should mention; swapping in a larger, hierarchical, or domain-specific concept set would directly test how much of the filtering gain comes from vocabulary coverage rather than from grounding quality.
The paper prunes noisy captions but does not edit them; a natural extension would be to combine OVFact scoring with caption rewriting or correction, using the metric to verify the edited output, which the authors themselves flag as future work.
Because the concept vocabulary is drawn from English-centric visual datasets, applying OVFact to multilingual or low-resource visual domains would likely require extending the vocabulary, and the paper's own limitation discussion notes that specialized domains need bespoke grounding tools.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The one thing to know: OVFact is a real step forward for measuring and improving long-caption factuality, but the headline filtering gains are partly self-fulfilling because the same metric selects and evaluates the data. The recall side is the soft spot.

What's actually new: combining LLM parsing with open-vocabulary detection and segmentation to get a reference-free precision-plus-recall score. That combination is new relative to CHAIR, THRONE, and ALOHa, and it enables the data-filtering application, which is the paper's most useful contribution. The human agreement study (Tab. 2/A1) is a legitimate external validation, and the fact that filtering also improves CHAIR (Fig. 5) is independent evidence that something real is happening. The 68.2% human preference for the model trained on 20% of ShareGPT4v is strong.

The soft spot: OVFactRec is defined against a fixed 2792-concept vocabulary, detected by OWL-ViT/OpenSeg, with grounding specificity of only 0.72–0.79. Descriptiveness is therefore measured against what the tools can recover, not against the actual scene. A caption mentioning a valid but out-of-vocabulary or too-fine-grained object gets no recall credit, and filtering on OVFactF1 can systematically discard such captions. The paper's own limitations section acknowledges this. The bigger issue is circularity: Fig. 4 evaluates filtering with the same OVFactF1 used for selection. The CHAIR and human-preference results mitigate but do not fully remove that concern. Also, the detection confidence threshold is never reported, so the metric is not fully reproducible.

Is it fatal? No. The paper is transparent about tool limitations, and the independent checks give real support. But a referee should push for: reporting the threshold and error bars, a descriptiveness measurement on DOCCI that does not rely on OVFactRec, and ideally an analysis of what OVFact-filtered data discards in terms of vocabulary coverage. The citation pattern looks solid; the authors build properly on prior work.

This paper deserves serious peer review. It's a well-posed problem, a sensible method, and the empirical work is substantial even if the central metric has a self-measurement component. I'd send it to review with requests for those fixes, and I'd bring it to a reading group: the circularity discussion is instructive.

Referee Report

3 major / 5 minor

Summary. This paper introduces OVFact, a reference-free metric for object-level factuality of long image captions. The method parses a caption into candidate entities with Gemma 2, grounds each entity in the image with OWL-ViTv2 and OpenSeg to compute precision (Eq. 1), and computes recall by comparing the caption entities against a reference set obtained by grounding 2,792 concepts from VG/LVIS/OpenImages/Objects365 through maximum cosine similarity (Eq. 2), then combines the two into an F1 score (Eq. 3). The metric is validated against human side-by-side preferences on DOCCI (Tabs. 2 and A1) and is then used to filter the ShareGPT4v dataset for fine-tuning PaliGemma 2 models. The authors report that models trained on the filtered subsets improve on OVFactF1 on DOCCI and Localized Narratives (Fig. 4), on CHAIR metrics on COCO (Fig. 5), and in a human preference study (68.2%, Sec. A.4), with 2.5-5x less training data.

Significance. If the results hold, OVFact is a useful practical contribution: it is a reference-free, open-vocabulary metric that unifies precision and recall for long captions, and the data-filtering application addresses a real need because VLM-generated pretraining captions are noisy. The paper deserves credit for the external validation of the precision axis via CHAIR (Fig. 5), the human-agreement analysis on both precision and recall (Sec. A.1), the efficiency analysis (Sec. A.7), and the use of openly available components. The main risk is that the recall/descriptiveness component, and hence a substantial part of the filtering result, is measured with the same scoring function used for selection; the independent anchors provide only partial coverage of the recall axis. The claimed 'without sacrificing descriptiveness' result therefore needs additional validation before the central contribution is fully established.

major comments (3)

[§3.2, §4.3, Fig. 4] OVFactF1 is used both as the selection score for filtering ShareGPT4v (Sec. 3.2) and as the primary downstream evaluation metric in Fig. 4, so the reported improvements are partly a selection effect on the target measure. The independent evidence in Fig. 5 covers precision-oriented hallucination on the 80 COCO classes, and the human study in Sec. A.4 asks about general quality rather than object-coverage recall. The abstract's claim that filtering improves factuality precision 'without sacrificing caption descriptiveness' is therefore not yet anchored by an independent recall measure. Please report the filtering comparison with a descriptiveness metric that is not OVFactRec (e.g., human-rated entity recall on a subset of each benchmark, or standard COCO recall), or otherwise show that the OVFactF1 gains persist when the evaluation axis is decoupled from the selection score.
[§3.1, Eq. (2), §4.1, Table 1] The recall term OVFactRec is defined against a fixed, tool-dependent reference set: 2,792 concepts from four datasets, grounded by OWL-ViTv2 and OpenSeg. Table 1 reports that only 0.72 (DOCCI) and 0.79 (Localized Narratives) of human-annotated entities are grounded, and the paper's own limitations section (Sec. 6) acknowledges that occluded, very fine-grained, or distant objects are often difficult for the grounding tools. As a result, captions that correctly mention valid but out-of-vocabulary or hard-to-ground objects receive systematically lower OVFactRec, and filtering on OVFactF1 can discard descriptive captions. Please quantify this bias by measuring OVFactRec against human-annotated entity recall per caption on a random sample and by varying the vocabulary V to show sensitivity; without this, the descriptiveness guarantee in the central claim is not established.
[Figs. 4 and 6] The filtering results in Figs. 4 and 6 are shown without error bars or significance tests, even though the text states that each curve is averaged over three training runs. Particularly at intermediate data ratios, the differences between methods could be within run-to-run variance. Please add standard deviations or confidence intervals to all curves and state whether the reported OVFactF1 improvements are statistically significant.

minor comments (5)

[Table 1] The label 'specificity' for entity grounding is misnamed: the quantity described, correctly grounded entities divided by all annotated entities, is a recall/sensitivity measure, not specificity.
[§4.3] The first paragraph refers to 'ShareGPTv'; this should be 'ShareGPT4v' for consistency with the rest of the paper.
[§3.1] The text says 'verifying the presence of associated concepts within image y' when y denotes the caption; the image variable is x elsewhere, so this notation should be corrected.
[§4.1 and Appendix A.5] The detection confidence threshold for OWL-ViTv2 and any threshold used for OpenSeg are not reported; these values are needed for reproducibility, since Eq. (1) depends directly on the threshold.
[Table A1] The per-pair human-agreement numbers appear to be based on small samples, and several entries (e.g., ALOHa at 25.0% for GPT4v x InstructBLIP) have no confidence intervals; adding counts or intervals would make the overall agreement claim more interpretable.

Circularity Check

2 steps flagged · score 6.0 of 10

Data-filtering gains on long-caption benchmarks are partly self-measured because the same OVFactF1 metric is used both to select the training subset and to evaluate downstream descriptiveness, though CHAIR and human evaluations provide independent partial support.

self definitional [Sec. 3.1, 'Calculating Recall', Eq. (2); Sec. 4.1, Implementation details]
"However, if the ground-truth captions are unreliable, e.g. for VLM-generated datasets, we also consider a general case to obtaining R by prompting the grounding tools given a large enough vocabulary of concepts V. ... OVFactRec = 1/|R| sum_{j=1}^{|R|} max_{i=1}^{|C|} S_ij (2)"

The reference-free recall estimate is not anchored to an external set of objects that a good caption should mention; R is, by construction, the set of concepts from the fixed 2792-concept vocabulary V that OWL-ViT and OpenSeg happen to ground. Descriptiveness is therefore measured as coverage of tool-detectable V concepts, not as coverage of actual image entities. Entities outside V, or too fine-grained, occluded, or otherwise missed by the grounding tools, receive no recall credit. The paper's own Tab. 1 reports entity-grounding specificity of only 0.72 on DOCCI and 0.79 on Localized Narratives, so the metric's 'descriptiveness' claim is self-defined by the tool/vocabulary pipeline rather than independently established.
fitted input called prediction [Sec. 3.2, 'OVFact for Data Filtering' vs. Sec. 4.3, 'OVFact for Improving Factuality', Fig. 4]
"Our data filtering approach consists of extracting OVFactF 1 scores for each (x, y) pair. ... We then sort the pairs based on their OVFact F 1 and select top X% depending on the assumed data pruning ratio. ... Fig. 4 presents the results of applying various data filtering methods and their downstream performance on DOCCI and Localized Narratives datasets, evaluated using our proposed metric (OVFactF 1)."

The training subset is selected by ranking ShareGPT4v pairs with OVFactF1, and the main long-caption downstream evidence in Fig. 4 is then measured with that same OVFactF1 metric on DOCCI and Localized Narratives. A model trained on data selected for high OVFactF1 is systematically biased to score well on OVFactF1, because the selection criterion and the evaluation axis share the same reference vocabulary, grounding tools, and matching procedure. Consequently, the claim that factuality precision improves 'without sacrificing caption descriptiveness' on these benchmarks is partly a self-fulfilling selection effect rather than an independent confirmation. The CHAIR results (Fig. 5) and the human side-by-side preference study (Sec.

full rationale

The paper's metric is validated against external human judgments and CHAIR, so it is not circular in its measurement-validation role: Tab. 2 and Table A1 show OVFactPrec/OVFactRec agree with human precision/descriptiveness preferences better than ALOHa, and Fig. 5 demonstrates improvements on the independent CHAIR benchmark. However, the contribution most emphasized in the abstract and conclusion—improved factuality 'without sacrificing caption descriptiveness across a range of downstream long caption benchmarks'—rests primarily on Fig. 4, where OVFactF1 is both the filter and the evaluation metric. That is a construction-level coupling: the selected data is scored by the exact metric used to measure the outcome, so the gains on DOCCI and Localized Narratives in Fig. 4 are partly forced by the selection procedure itself. In addition, the reference-free recall definition measures descriptiveness against a fixed tool-grounded concept inventory V rather than against the image's actual content, making the descriptor claim partly self-definitional. These two issues warrant a 6 rather than a higher score because independent evidence (CHAIR, human preference, model-size studies, and per-component specificity analyses) prevents the whole contribution from reducing to a single self-measurement.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The metric rests on two proxies: an LLM parser that extracts entities and two grounding tools that verify them. Neither is a ground-truth oracle; the paper reports parser specificity 97% and grounding specificity 0.72-0.79 on small samples. The recall branch additionally depends on a fixed, hand-assembled concept vocabulary, which bounds the claimed open-vocabulary property. The detection confidence threshold is unreported, adding an uncontrolled degree of freedom.

free parameters (2)

Detection confidence threshold
In Sec. 3.1 (Entity grounding), a candidate entity is grounded if its detection confidence score exceeds a threshold value; the threshold is never reported or justified, and it directly controls OVFactPrec and hence the F1 score.
Reference vocabulary V (2792 concepts) = 2792 concepts
The recall term uses a fixed, hand-assembled union of concepts from Visual Genome, LVIS, Open Images and Objects365; true open-vocabulary recall is not measured, only recall within V.

assumptions (3)

domain assumption Gemma2-27b parsing extracts all visually present objects from captions with high specificity
Section 4.2, Tab. 1 reports 97% specificity on 100 DOCCI and 100 Localized Narratives samples; the metric's candidate set C depends on this.
domain assumption OWL-ViT and OpenSeg can ground any visually present object described in the caption
Section 3.1 builds G = GD ∪ GS on these tools; the paper acknowledges in Limitations that occluded, fine-grained or distant objects are difficult to ground.
ad hoc to paper The fixed 2792-concept vocabulary V approximates the space of all visual concepts relevant to long captions
Section 4.1 states V is the union of concepts from four datasets; this closed set contradicts the 'open-vocabulary' framing for recall and determines OVFactRec.

how reviews work

0 comments

Cite this review

Pith. "Pith review of OVFact: Measuring and Improving Open-Vocabulary Factuality for Long Caption Models." pith.science (2026). https://pith.science/paper/7OZD5FUC

@misc{pith2026250719262,
  author       = {Pith},
  title        = {Pith review of: OVFact: Measuring and Improving Open-Vocabulary Factuality for Long Caption Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7OZD5FUC}},
  note         = {Machine review of arXiv:2507.19262}
}

read the original abstract

Large vision-language models (VLMs) often struggle to generate long and factual captions. However, traditional measures for hallucination and factuality are not well suited for evaluating longer, more diverse captions and in settings where ground-truth human-annotated captions are unavailable. We introduce OV-Fact, a novel method for measuring caption factuality of long captions that leverages open-vocabulary visual grounding and tool-based verification without depending on human annotations. Our method improves agreement with human judgments and captures both caption descriptiveness (recall) and factual precision in the same metric. Furthermore, unlike previous metrics, our reference-free method design enables new applications towards factuality-based data filtering. We observe models trained on an OVFact-filtered (2.5-5x less) subset of a large-scale, noisy (VLM-generated) pretraining set meaningfully improve factuality precision without sacrificing caption descriptiveness across a range of downstream long caption benchmarks.

Figures

Figures reproduced from arXiv: 2507.19262 by the authors.

**Figure 1.** Measuring and improving factuality. We propose OVFact, a method for measuring and improving open-vocabulary factuality of vision-language models (VLMs) in long captioning. (top) Unlike prior work tailored for short captions or specific datasets, OVFact is reference-free and does not require annotated groundtruth. (bottom) Our flexible design enables a new factuality-based data filtering application, where models t… view at source ↗

**Figure 2.** OVFact overview. Our reference-free method assesses two aspects of long captioning – precision and descriptiveness (recall) – in a unified manner, without requiring ground-truth reference annotations. We process a model output caption into a set of candidate entities C, and assess which subset G are groundable in the input image with open-vocabulary detection and segmentation tools. Precision is then measured as a r… view at source ↗

**Figure 3.** Limitations of ALOHa in a reference-free setting. We present an example from ShareGPT4v (i.e., with no access to ground-truth references). ALOHa’s reference matching forces comparisons of detected classes with what has been described, an issue exacerbated in a reference-free setting when ground-truth human annotations of objects are unavailable. Our method is more robust to such cases. ing (Kuhn, 1955) on the simi… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Data filtering comparison measured with OVFactF 1. We present the results of ShareGPT4v filtering for different data ratios, with zero-shot evaluations on DOCCI test and Localized Narratives val sets averaged over 3 training runs. ’---’ indicates model trained on full …

**Figure 5.** Figure 5: Generalization to CHAIR benchmark. We also measure zero-shot evaluation on the CHAIR benchmark, highlighting how our filtering improves on traditional metrics and settings too. Light-blue shading marks variance of the random baseline. Our filtering method results in fe…

**Figure 6.** Figure 6: Data-filtering ablation study. We train PaliGemma 2 on filtered subsets of ShareGPT4V (with ablations of our method), and evaluate models zero-shot on the DOCCI test set. Note, top-right corner is the best. Filtering with full OVFact (purple) yields models with best tr…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 9 canonical work pages

[1]

Include all visual attributes and adjectives that describe the object, if present

work page
[2]

arXiv preprint arXiv:2407.07726

PaliGemma: A versatile 3B VLM for transfer. arXiv preprint arXiv:2407.07726. Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov

arXiv
[3]

Always give the singular form of the object, even if the caption uses the plural form

work page
[4]

late fusion

Return only a python list. I will give you some examples. Image caption: A cozy room illuminated by a warm fireplace and soft candles. A large painting hangs on the wall, while a small painting adorns the opposite side of the fireplace. A wooden desk sits in the center of the room, with a book open on its surface. A white candle burns brightly on the desk...

work page 2024
[5]

arXiv preprint arXiv:2309.17425

Data filtering networks. arXiv preprint arXiv:2309.17425. Alessandro Favero, Luca Zancato, Matthew Trager, Sid- dharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto

arXiv
[6]

Multi-modal hallucination control by visual information grounding. In CVPR. David A Forsyth, Jitendra Malik, Margaret M Fleck, Hayit Greenspan, Thomas Leung, Serge Belongie, Chad Carson, and Chris Bregler. 1996. Finding pic- tures of objects in large collections of images. In ECCV. Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios...

arXiv 1996
[7]

Scaling open-vocabulary image segmentation with image-level labels. In ECCV. Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2024. Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vis...

work page 2024
[8]

In ACM Multimedia 2024

Hal-eval: A universal and fine-grained hallu- cination evaluation framework for large vision lan- guage models. In ACM Multimedia 2024. Andrej Karpathy and Li Fei-Fei. 2015. Deep visual- semantic alignments for generating image descrip- tions. In CVPR. Prannay Kaul, Zhizhong Li, Hao Yang, Yonatan Duk- ler, Ashwin Swaminathan, CJ Taylor, and Stefano Soatto...

work page 2024

Show all 19 references

[10]

The role of data curation in image captioning. In ACL. 11 Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. In EMNLP. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Per...

2023 arXiv
[11]

Scaling open-vocabulary object detection. NeurIPS. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the conference on f...

2019 arXiv
[12]

Valor-eval: Holistic coverage and faithfulness evaluation of large vision-language models. In ACL. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transf...

2021
[15]

models with input images of size 448 x 448, following the best practices outlined in the orig- inal paper. Our main experiments in the paper focus on the PaliGemma 2 model with 3B param- eters (SigLIP-So400m encoder with 14x14 pixel patches, yielding 1024 tokens per image, and...

2024
[17]

Do not include objects that are mentioned but have no visual presence in the image, such as light, sound, or emotions
[2017]

Visual genome: Connecting language and vi- sion using crowdsourced dense image annotations. IJCV. Harold W. Kuhn. 1955. Naval Research Logistics Quar- terly. Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Pop...

1955
[2019]

Objects365: A large-scale, high-quality dataset for object detection. In ICCV. Richard Socher, Andrej Karpathy, Quoc V Le, Christo- pher D Manning, and Andrew Y Ng. 2014. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Ass...

2014 arXiv
[2022]

https://github.com/ google-research/big_vision

Big vision. https://github.com/ google-research/big_vision. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosse- lut, Emma Brunskill, et al. 2021. On the opportuni- ties and risks of foundat...

2021 arXiv
[2023]

arXiv preprint arXiv:2307.06290

Instruction mining: Instruction data selection for tuning large language models. arXiv preprint arXiv:2307.06290. Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. ...

2020 arXiv
[2024]

arXiv preprint arXiv:2404.18930

Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V . Do, Yan Xu, and Pascale Fung. 2023. A mult...

2023 arXiv
[2025]

brown white fur

One slice is not enough: In search of stable conclusions in text-to-image evaluation. In ICLR. Xiyang Wu, Tianrui Guan, Dianqi Li, Shuaiyi Huang, Xiaoyu Liu, Xijun Wang, Ruiqi Xian, Abhinav Shri- vastava, Furong Huang, Jordan Lee Boyd-Graber, Tianyi Zhou, and Dinesh Manocha. 2...

2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Include all visual attributes and adjectives that describe the object, if present

work page

[2] [2]

arXiv preprint arXiv:2407.07726

PaliGemma: A versatile 3B VLM for transfer. arXiv preprint arXiv:2407.07726. Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov

arXiv

[3] [3]

Always give the singular form of the object, even if the caption uses the plural form

work page

[4] [4]

late fusion

Return only a python list. I will give you some examples. Image caption: A cozy room illuminated by a warm fireplace and soft candles. A large painting hangs on the wall, while a small painting adorns the opposite side of the fireplace. A wooden desk sits in the center of the room, with a book open on its surface. A white candle burns brightly on the desk...

work page 2024

[5] [5]

arXiv preprint arXiv:2309.17425

Data filtering networks. arXiv preprint arXiv:2309.17425. Alessandro Favero, Luca Zancato, Matthew Trager, Sid- dharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto

arXiv

[6] [6]

Multi-modal hallucination control by visual information grounding. In CVPR. David A Forsyth, Jitendra Malik, Margaret M Fleck, Hayit Greenspan, Thomas Leung, Serge Belongie, Chad Carson, and Chris Bregler. 1996. Finding pic- tures of objects in large collections of images. In ECCV. Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios...

arXiv 1996

[7] [7]

Scaling open-vocabulary image segmentation with image-level labels. In ECCV. Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2024. Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vis...

work page 2024

[8] [8]

In ACM Multimedia 2024

Hal-eval: A universal and fine-grained hallu- cination evaluation framework for large vision lan- guage models. In ACM Multimedia 2024. Andrej Karpathy and Li Fei-Fei. 2015. Deep visual- semantic alignments for generating image descrip- tions. In CVPR. Prannay Kaul, Zhizhong Li, Hao Yang, Yonatan Duk- ler, Ashwin Swaminathan, CJ Taylor, and Stefano Soatto...

work page 2024