Pith. sign in

REVIEW 5 major objections 7 minor 100 references

Understanding Museum Exhibits using Vision-Language Reasoning

T0 review · 5 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 65M-image, 200M-question museum dataset lifts fine-tuned vision-language models far above zero-shot generalists on exhibit question answering.

desk verdict A genuinely large and well-curated museum VQA dataset with an overclaimed in-distribution evaluation; worth refereeing if the promised release actually happens. read the letter →

arxiv 2412.01370 v2 pith:5EVK6WIP submitted 2024-12-02 cs.CV cs.CL

Ada-Astrid Balauca , Sanjana Garai , Stefan Balauca , Rasesh Udayakumar Shetty , Naitik Agrawal , Dhwanil Subhashbhai Shah , Yuqian Fu , Xi Wang
show 3 more authors
Kristina Toutanova Danda Pani Paudel Luc Van Gool
This is my paper · ORCID
classification cs.CVcs.CL
keywords visualquestionansweringvision-languagemodelsmuseumexhibitsculturalheritagefine-tuningmultimodaldatasetmultilingualVQAmulti-angleevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that museum exhibits can be understood by vision-language models if the models are fine-tuned on a sufficiently large, expert-curated dataset. It introduces MUSEUM-65, a collection of 65 million exhibit images paired with 200 million question-answer pairs, and trains two model families on subsets of it. Across five visual-question-answering tasks, the fine-tuned models substantially outperform zero-shot state-of-the-art VLMs on attribute questions such as title, place of origin, and material. The paper also reports that LLaVA, the instruction-tuned model with a larger language backbone, outperforms BLIP on questions requiring historical context and reasoning, while both models handle visually grounded questions well. If correct, this establishes domain-specific large-scale fine-tuning as a practical route to museum AI applications.

What carries the argument

The load-bearing mechanism is MUSEUM-65 itself: museum metadata scraped from three international aggregators and twelve additional museums is cleaned into attribute-value pairs, then 63 manually crafted question templates turn each pair into a QA instance with the attribute value as answer. Fine-tuning BLIP, an encoder-decoder VLM with vision-language aligned embeddings, and LLaVA, an instruction-tuned large language model with vision-language reasoning, on subsets of 1, 10, or 20 million samples supplies the comparison. The five-task benchmark, covering general VQA, category-wise VQA, multi-angle robustness, visually unanswerable questions, and multilingual queries, is what converts raw accuracy into claims about viewpoint robustness, reasoning, and language transfer.

What would settle it

Evaluate the fine-tuned models on a test set of questions written by museum visitors or domain experts that share no templates with the 63 training questions; if performance falls to near the zero-shot baseline while in-distribution scores stay high, the reported gains are template memorization rather than museum understanding.

Watch

Extended reading notes

Core claim

The central claim is that a large, structured, expert-labeled dataset of museum metadata, reformulated into natural-language questions, is sufficient to make existing vision-language models perform at a level far above zero-shot generalists on exhibit understanding. The evidence: BLIP fine-tuned on 20 million samples reaches 71.51% partial precision and 60.58% complete precision on general VQA, while LLaVA fine-tuned on 20 million samples for one epoch reaches 81.25% partial precision and 63.96% complete precision, with zero-shot SOTA models scoring in the low 20s to 40s on the same attribute questions. On the manually built Visually Unanswerable set, LLaVA reaches 29.7% partial precision on unseen images while BLIP drops to 3.3%. The enabling factor is the dataset's attribute-value structure: museum metadata is converted by human-written templates into 200 million QA pairs, giving models repeated exposure to the exact vocabulary and reasoning patterns of exhibit catalogues.

Load-bearing premise

The held-out test questions are generated by the same 63 templates and the same metadata pipeline as the training data, so the central claim assumes that high scores on this in-distribution test reflect genuine museum understanding rather than memorized template-answer patterns.

Editorial extensions

If this is right

  • Fine-tuned VLMs can support museum visitor question answering for attribute-level queries such as title, material, and place of origin at much higher accuracy than general-purpose zero-shot models.
  • Instruction-tuned, LLM-based VLMs are the more promising architecture when questions require historical context or reasoning beyond what is visible in the image.
  • Domain-specific large-scale fine-tuning is necessary for strong performance on nuanced museum attributes, since zero-shot SOTA models lag far behind the fine-tuned models.
  • The dataset and benchmark provide a reusable testbed for five VQA tasks, including multilingual queries and viewpoint changes, which are directly relevant to real museum deployments.
  • Cross-evaluation on the MUZE benchmark suggests that models trained on MUSEUM-65 can transfer to existing cultural-heritage datasets and outperform models trained on those datasets themselves.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next check the paper leaves implicit is whether the gains survive on naturally occurring visitor questions whose phrasing was not drawn from the 63 templates; if they do not, the reported scores would mostly reflect template memorization.
  • The multilingual results suggest a concrete extension: fine-tuning on the 15 million non-English samples in MUSEUM-65 could reverse the language forgetting observed in the English-only LLaVA model.
  • Because the dataset releases raw metadata alongside the curated QA pairs, it could support retrieval-based and knowledge-augmented VQA systems that answer visually unanswerable questions with external references.
  • The attribute-value structure also invites a diagnostic use beyond training: probing which attribute categories remain hard after fine-tuning, such as title and subject, could guide where museum metadata needs to be richer.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. This paper introduces MUSEUM-65, a large-scale multilingual dataset of museum exhibit images paired with question-answer pairs generated from museum metadata, and fine-tunes BLIP and LLaVA models on subsets of it. The authors propose five VQA tasks: general VQA, category-wise VQA, multi-angle VQA, visually unanswerable questions, and multilingual VQA, and report that fine-tuned models outperform zero-shot state-of-the-art models on attribute questions, with the fine-tuned LLaVA model performing best. They also report a cross-evaluation on MUZE, a comparison with human experts, and an ablation on place of origin.

Significance. The paper's main potential contribution is the scale and public availability of the dataset, and the MUZE cross-evaluation in Table 17 provides a useful external check that training on MUSEUM-65 transfers to another benchmark. The manually constructed Visually Unanswerable set is a genuine attempt to go beyond purely visual grounding. However, the central benchmarking claim is weakened by the in-distribution construction of the test set, and the 'reasoning' claim rests on a small test set with modest partial-precision gains. The dataset could still be a valuable resource if the authors restructure the evaluation to include out-of-distribution generalization checks and temper the claims accordingly.

major comments (5)
  1. [Sec. 3.2, 3.5, App. B.6 (Table 12)] The main benchmark is in-distribution by construction: both training and test questions are generated from the same 63 hand-written templates applied to the same attribute-value metadata, and Sec. 3.5 does not describe any object-level or template-level split. Consequently, the gains reported in Table 1 and Table 18 may largely reflect learning the template-to-metadata mapping and the exact output formatting rather than understanding museum exhibits from images. Please enforce object-level and template-level separation (e.g., hold out entire question templates, or at least verify that no object appears in both train and test), and report results on such splits, or restrict the headline claims to the manually curated Task 4 and the MUZE cross-evaluation in Table 17.
  2. [Sec. 4.1, Table 18] The definition of 'Complete Precision' in Sec. 4.1 is the fraction of questions with precision=1.0, which only requires that every word in the prediction appears in at least one valid answer; it does not require the prediction to cover the full ground-truth answer. Under this metric, a short correct substring scores 100%, so the comparison between zero-shot (verbose) and fine-tuned (terse) models is inflated by output length and phrasing differences. Please report exact-match accuracy and recall-based metrics alongside, or justify why the defined metric is appropriate for the cross-model comparison.
  3. [Sec. 5.5, Tables 5 and 6] The claim that large VLMs 'excel in queries requiring deeper historical context and reasoning' (abstract) rests on 515 test pairs in Task 4. On these pairs, LLaVA20mn-1ep reaches 29.7% partial precision versus 27.18% for LLaVA without fine-tuning and 25.83% complete precision versus 1.55%, with no confidence intervals or significance tests. The partial-precision gap is small, and many of these questions ask for world knowledge about the object's country of origin, an attribute on which the model was fine-tuned. Please provide bootstrap confidence intervals, a breakdown by question type, and an analysis of failure cases to substantiate the reasoning interpretation.
  4. [Sec. 5.3, Fig. 6] The statement that fine-tuned models 'outperform human experts across all categories' is based on 850 QA pairs answered by 10 experts and scored against museum metadata strings with the same lexical metrics. Humans were not given the metadata source, and the metric rewards exact string overlap, so this comparison conflates metadata retrieval with domain expertise. The IAA paragraph reports 52.7% agreement and a 6.25% chance baseline, but the number of categories and the simulation procedure are not described. Please provide the full protocol or soften the claim.
  5. [Abstract and Sec. 3.2] The abstract's statement that 'the complete dataset is labeled by museum experts' is not supported by the curation description in Sec. 3.2, where experts clean existing museum metadata and manually write 63 question templates; the per-item answers are museum metadata values, not expert-authored labels. Please revise the wording to describe the annotation process accurately.
minor comments (7)
  1. [Table 1] The model names 'LLaVa nofinetune' and 'LLaVa-ours' are inconsistently capitalized; please use one consistent name (e.g., LLaVA) throughout.
  2. [Tables 5 and 6] The rows 'LLaV A 20mm-1ep' and 'BLIP 20mm-5ep' contain the typo '20mm'; these should be '20mn'.
  3. [Fig. 10 caption] The caption refers to 'LLaV A20mn-5ep', but the main text's best model is LLaVA20mn-1ep; please clarify which model produced the examples.
  4. [App. B.4] The sentence 'The links to the curated and raw datasets can be found here: MUSEUM-65' does not display an actual URL; please include the full link in the manuscript.
  5. [Sec. 4.1] The metric names 'Complete Precision' and 'Partial Precision' are misleading because they are not exact-match fractions; consider renaming them to something like 'No-False-Positive Precision' and 'Any-Hit Precision' to avoid confusion.
  6. [App. B.5] The section heading 'Datasest Curation Process' contains a typo; it should be 'Dataset Curation Process'.
  7. [Sec. 3.3, Table 8] The claim that 'benefits distribute evenly' across continents is supported only by point estimates in Table 8; no confidence intervals or significance tests are provided, so the strength of this claim is overstated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical benchmark results, and the main comparisons are grounded in held-out test data and an external MUZE cross-evaluation.

full rationale

The paper is an empirical dataset-and-benchmark contribution, not a derivation. The central comparisons, such as fine-tuned BLIP/LLaVA versus zero-shot SOTA (Table 1) and general VQA results (Table 18), are supervised evaluations on a held-out test split described in Section 3.5; the reported numbers are therefore not equivalent to the training objective by construction. The in-distribution nature of the 63-template benchmark (Section 3.2, Appendix B.6) is a real limitation for the broader understanding interpretation, but it is not circularity: the test answers are not derived from the fitted parameters, and the model must still map novel images to metadata values. The strongest qualitative claim about reasoning is based on the manually constructed Visually Unanswerable task (Section 5.5), which uses external general-knowledge answers and is evaluated on a separate 515-pair test set, providing independent content. The self-citation to MUZE [5] is used for a cross-dataset comparison (Appendix D.1) where the external MUZE dataset and a MUZE-trained BLIP baseline provide an independent benchmark; the citation is not load-bearing and, per the review rules, does not raise the circularity score. No fitted parameter is renamed as a prediction, and no uniqueness or ansatz is imported from prior work.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central result rests on dataset construction choices (templates, attribute filtering, splits) rather than on fitted physical parameters; the most consequential choices are the 63 question templates and the 100-word attribute cutoff, which define the distribution the models are trained and evaluated on.

free parameters (4)
  • Question template set size = 63 unique templates
    Manually authored by experts (App B.5, Table 12); determines all training and test questions, so answer distributions are constrained to metadata fields.
  • Medium attribute length threshold = 100 words
    Chosen to fit BLIP's 512-token limit (Sec 3.2); long attributes excluded from training and test, shaping task difficulty.
  • Training subset sizes = 1M, 10M, 20M instances
    Compute-driven choices (Sec 3.5, App B.3); best model uses 20M, not the full 42M or 65M.
  • Epochs and LoRA rank = 1-5 epochs, LoRA as in LLaVA
    Selected by monitoring validation (App C.3, Table 16); affects reported accuracies but not the qualitative conclusion.
assumptions (5)
  • domain assumption Museum metadata scraped from aggregators (DPLA, Europeana, Smithsonian) is accurate ground truth for object attributes.
    Questions are generated from these metadata values and treated as correct answers (Sec 3.2). If metadata contains errors, all labels inherit them.
  • ad hoc to paper The 63 hand-written question templates (Table 12) span the range of real visitor queries in museums.
    No evidence that visitor questions match these templates; phrasing diversity is from experts, not from observed museum dialogue (App B.5).
  • domain assumption Held-out test splits from the same museums and metadata pipelines measure generalization for museum VQA.
    Test set is generated by the same templates as training (Sec 3.5, App B.6), so in-distribution gains are expected; external MUZE evaluation (App D.1) partially addresses this.
  • domain assumption CLIP-based vision encoders in BLIP and LLaVA provide adequate visual features for artifact recognition.
    Both models inherit CLIP embeddings (Sec B.9); failures on visual reasoning are attributed to model capacity, not encoder.
  • domain assumption Fine-tuning on a random subset (up to 20M of 42M English instances) preserves the properties of the full dataset.
    Training uses 1M, 10M, or 20M subsets; claims about dataset scale rely on these subsets being representative.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Museum Exhibits using Vision-Language Reasoning." pith.science (2026). https://pith.science/paper/5EVK6WIP

@misc{pith2026241201370,
  author       = {Pith},
  title        = {Pith review of: Understanding Museum Exhibits using Vision-Language Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5EVK6WIP}},
  note         = {Machine review of arXiv:2412.01370}
}
read the original abstract

Museums serve as repositories of cultural heritage and historical artifacts from diverse epochs, civilizations, and regions, preserving well-documented collections that encapsulate vast knowledge, which, when systematically structured into large-scale datasets, can train specialized models. Visitors engage with exhibits through curiosity and questions, making expert domain-specific models essential for interactive query resolution and gaining historical insights. Understanding exhibits from images requires analyzing visual features and linking them to historical knowledge to derive meaningful correlations. We facilitate such reasoning by (a) collecting and curating a large-scale dataset of 65M images and 200M question-answer pairs for exhibits from all around the world; (b) training large vision-language models (VLMs) on the collected dataset; (c) benchmarking their ability on five visual question answering tasks, specifically designed to reflect real-world inquiries and challenges observed in museum settings. The complete dataset is labeled by museum experts, ensuring the quality and the practical significance of the labels. We train two VLMs from different categories: BLIP with vision-language aligned embeddings, but lacking the expressive power of large language models, and the LLaVA model, a powerful instruction-tuned LLM enriched with vision-language reasoning capabilities. Through extensive experiments, we find that while both model types effectively answer visually grounded questions, large vision-language models excel in queries requiring deeper historical context and reasoning. We further demonstrate the necessity of fine-tuning models on large-scale domain-specific datasets by showing that our fine-tuned models significantly outperform current SOTA VLMs in answering questions related to specific attributes, highlighting their limitations in handling complex, nuanced queries.

Figures

Figures reproduced from arXiv: 2412.01370 by the authors.

Figure 1
Figure 1. Dataset composition. MUSEUM-65 covers a wide range of exhibit categories (top), e.g arts, historical/pre-historical, natural sciences, and contains a large number of images from around the globe. Each image is paired with multiple questions exploring subjects like Title, Creator, Period, Techniques, Culture, Inscriptions, etc. (right). A sample image with a question and answer is shown on the left. viewpoints, Visua… view at source ↗
Figure 2
Figure 2. Dataset statistics. (a) distribution of questions, categorized by type: the most common question is about the objects’ place of origin, (b) number of distinct values of each category: the most varied category is subject (c) data sources of each contributing museum, (d) the most common subjects/tags associated with the exhibits: objects coming from historical museums, like maps, items related to the United States, or… view at source ↗
Figure 3
Figure 3. Workflow. Using smaller subsets of the dataset (1mn, 10mn and 20mn), we fine-tune BLIP and LLaVA models. BLIP, an encoder-decoder based model, aligns language and image in the same space while LLaVA, built on an instruction-tuned LLM is directly reasons based on the language. is the usual format for museum exhibit information. Each museum has a unique set of attributes. After extracting the attributes, we reformulat… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Benchmarked tasks. (1) general VQA, (2) category-wise VQA, (3) MultiAngle - measures the adaptability to different angle images of the same object, (4) Visually Unanswerable Questions - observes the response to new common knowledge questions derived from dataset’s avai…
Figure 5
Figure 5. Figure 5: General VQA results. Comparison of fine-tuned and non-fine-tuned models on precision and recall. Models fine￾tuned with the 20mn dataset perform best, with LLaVA20mn￾1ep achieving 80% partial precision and 64% complete precision. LLaVA models also outperform BLIP in re…
Figure 7
Figure 7. Figure 7: Examples. LLaVA20mn-1ep and BLIP20mn-5ep behaviour on different tasks, General VQA (1st column), MultiAngle (2nd column), Visually Unanswerable Questions (3rd column) and MultiLanguage (4th column). We observe more precise answers for LLaVA20mn than for BLIP20mn on all…
Figure 8
Figure 8. Figure 8: Comparison between BLIP20mn-1ep ad BLIP1mn-20ep [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: VQA category-wise results. On left be compared all BLIP models and in right all LLaVA models. The fine-tuned models do better on all categories. The original ones only per￾form well for language and collection, as they have easier, com￾mon knowledge answers (for collec…
Figure 10
Figure 10. Figure 10: Examples of LLaVA20mn-5ep results for the proposed tasks. The question is denoted with (Q), the answer wit (A) and the [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

100 extracted references · 55 canonical work pages

  1. [1]

    Artemis: Affective language for visual art

    Panos Achlioptas, Maks Ovsjanikov, Kilichbek Haydarov, Mohamed Elhoseiny, and Leonidas J Guibas. Artemis: Affective language for visual art. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11569–11579, 2021. 3

    2021

  2. [2]

    Feelingblue: A corpus for understanding the emotional con- notation of color in context.Transactions of the Association for Computational Linguistics, 11:176–190, 2023

    Amith Ananthram, Olivia Winn, and Smaranda Muresan. Feelingblue: A corpus for understanding the emotional con- notation of color in context.Transactions of the Association for Computational Linguistics, 11:176–190, 2023. 3

    2023

  3. [3]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. InProceedings of the IEEE international conference on computer vision, pages 2425– 2433, 2015. 1

    2015

  4. [4]

    Explain me the painting: Multi-topic knowledgeable art description gen- eration

    Zechen Bai, Yuta Nakashima, and Noa Garcia. Explain me the painting: Multi-topic knowledgeable art description gen- eration. InProceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 5422–5432, 2021. 3

    2021

  5. [5]

    Taming CLIP for Fine-grained and Structured Visual Understanding of Museum Exhibits

    Ada-Astrid Balauca, Danda Pani Paudel, Kristina Toutanova, and Luc Van Gool. Taming clip for fine-grained and struc- tured visual understanding of museum exhibits.arXiv preprint arXiv:2409.01690, 2024. 2, 3

    work page Pith review arXiv 2024

  6. [6]

    Bridg- ing the gap between object and image-level representations for open-vocabulary detection.Advances in Neural Informa- tion Processing Systems, 35:33781–33794, 2022

    Hanoona Bangalath, Muhammad Maaz, Muhammad Uzair Khattak, Salman H Khan, and Fahad Shahbaz Khan. Bridg- ing the gap between object and image-level representations for open-vocabulary detection.Advances in Neural Informa- tion Processing Systems, 35:33781–33794, 2022. 1, 2

    2022

  7. [7]

    Clip retrieval: Easily compute clip embeddings and build a clip retrieval system with them.https : / / github

    Romain Beaumont. Clip retrieval: Easily compute clip embeddings and build a clip retrieval system with them.https : / / github . com / rom1504 / clip - retrieval, 2022. 16

    2022

  8. [8]

    Viscounth: A large-scale multilin- gual visual question answering dataset for cultural heritage

    Federico Becattini, Pietro Bongini, Luana Bulla, Al- berto Del Bimbo, Ludovica Marinucci, Misael Mongiov `ı, and Valentina Presutti. Viscounth: A large-scale multilin- gual visual question answering dataset for cultural heritage. ACM Transactions on Multimedia Computing, Communica- tions and Applications, 2023. 1, 2, 3

    2023

Show all 100 references
  1. [9]

    Predicting image aesthetics with deep learning

    Simone Bianco, Luigi Celona, Paolo Napoletano, and Rai- mondo Schettini. Predicting image aesthetics with deep learning. InAdvanced Concepts for Intelligent Vision Sys- tems: 17th International Conference, ACIVS 2016, Lecce, Italy, October 24-27, 2016, Proceedings 17, pages 11...

    2016

  2. [10]

    Vizwiz: nearly real-time answers to visual questions

    Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Lit- tle, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. InProceedings of the 23nd annual ACM symposium on User inter...

    2010

  3. [11]

    Visual question answering for cul- tural heritage

    Pietro Bongini, Federico Becattini, Andrew D Bagdanov, and Alberto Del Bimbo. Visual question answering for cul- tural heritage. InIOP Conference Series: Materials Science and Engineering, page 012074. IOP Publishing, 2020. 3

    2020

  4. [12]

    Fine-tuning convolutional neural networks for fine art classification.Ex- pert Systems with Applications, 114:107–118, 2018

    Eva Cetinic, Tomislav Lipic, and Sonja Grgic. Fine-tuning convolutional neural networks for fine art classification.Ex- pert Systems with Applications, 114:107–118, 2018. 3

    2018

  5. [13]

    Uniter: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. InEuropean conference on computer vision, pages 104–120. Springer,

  6. [14]

    Clip-art: Contrastive pre-training for fine-grained art classification

    Marcos V Conde and Kerem Turgutlu. Clip-art: Contrastive pre-training for fine-grained art classification. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3956–3960, 2021. 2

    2021

  7. [15]

    Learning sample difficulty from pre-trained models for reliable prediction.Advances in Neural Information Process- ing Systems, 36, 2024

    Peng Cui, Dan Zhang, Zhijie Deng, Yinpeng Dong, and Jun Zhu. Learning sample difficulty from pre-trained models for reliable prediction.Advances in Neural Information Process- ing Systems, 36, 2024. 1

    2024

  8. [16]

    Novel datasets for fine-grained image categoriza- tion

    E Dataset. Novel datasets for fine-grained image categoriza- tion. InFirst Workshop on Fine Grained Visual Categoriza- tion, CVPR. Citeseer. Citeseer. Citeseer, 2011. 3

    2011

  9. [17]

    Noisyart: A dataset for webly-supervised art- work recognition

    Riccardo Del Chiaro, Andrew D Bagdanov, and Alberto Del Bimbo. Noisyart: A dataset for webly-supervised art- work recognition. InVISIGRAPP (4: VISAPP), pages 467– 475, 2019. 3

    2019

  10. [18]

    Webly-supervised zero-shot learning for artwork instance recognition.Pattern Recognition Letters, 128:420– 426, 2019

    Riccardo Del Chiaro, Andrew D Bagdanov, and Alberto Del Bimbo. Webly-supervised zero-shot learning for artwork instance recognition.Pattern Recognition Letters, 128:420– 426, 2019. 3

    2019

  11. [19]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 3

    2009

  12. [20]

    Stytr2: Im- age style transfer with transformers

    Yingying Deng, Fan Tang, Weiming Dong, Chongyang Ma, Xingjia Pan, Lei Wang, and Changsheng Xu. Stytr2: Im- age style transfer with transformers. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11326–11336, 2022. 1

    2022

  13. [21]

    De- coupling zero-shot semantic segmentation

    Jian Ding, Nan Xue, Guisong Xia, and Dengxin Dai. De- coupling zero-shot semantic segmentation. 2022 ieee. In CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), pages 11573–11582, 2021. 1, 2

    2022

  14. [22]

    A survey on bias in visual datasets.Computer Vision and Image Understanding, 223: 103552, 2022

    Simone Fabbrizzi, Symeon Papadopoulos, Eirini Ntoutsi, and Ioannis Kompatsiaris. A survey on bias in visual datasets.Computer Vision and Image Understanding, 223: 103552, 2022. 4, 16

    2022

  15. [23]

    Are you talking to a machine? dataset and methods for multilingual image question.Advances in neural information processing systems, 28, 2015

    Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu. Are you talking to a machine? dataset and methods for multilingual image question.Advances in neural information processing systems, 28, 2015. 3

    2015

  16. [24]

    How to read paintings: semantic art understanding with multi-modal retrieval

    Noa Garcia and George V ogiatzis. How to read paintings: semantic art understanding with multi-modal retrieval. In Proceedings of the European Conference on Computer Vi- sion (ECCV) Workshops, pages 0–0, 2018. 3

    2018

  17. [25]

    Knowit vqa: Answering knowledge-based ques- tions about videos

    Noa Garcia, Mayu Otani, Chenhui Chu, and Yuta Nakashima. Knowit vqa: Answering knowledge-based ques- tions about videos. InProceedings of the AAAI conference on artificial intelligence, pages 10826–10834, 2020. 3

    2020

  18. [26]

    A dataset and baselines for visual question answering on art

    Noa Garcia, Chentao Ye, Zihua Liu, Qingtao Hu, Mayu Otani, Chenhui Chu, Yuta Nakashima, and Teruko Mita- mura. A dataset and baselines for visual question answering on art. InComputer Vision–ECCV 2020 Workshops: Glas- gow, UK, August 23–28, 2020, Proceedings, Part II 16, pages...

    2020

  19. [27]

    A dataset and baselines for visual question answering on art

    Noa Garcia, Chentao Ye, Zihua Liu, Qingtao Hu, Mayu Otani, Chenhui Chu, Yuta Nakashima, and Teruko Mita- mura. A dataset and baselines for visual question answering on art. InComputer Vision–ECCV 2020 Workshops: Glas- gow, UK, August 23–28, 2020, Proceedings, Part II 16, pages...

    2020

  20. [28]

    Im- age style transfer using convolutional neural networks

    Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Im- age style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414–2423, 2016. 1

    2016

  21. [29]

    Aes- thetic image captioning from weakly-labelled photographs

    Koustav Ghosal, Aakanksha Rana, and Aljosa Smolic. Aes- thetic image captioning from weakly-labelled photographs. InProceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019. 3

    2019

  22. [30]

    Beyond language bias: Over- coming multimodal shortcut and distribution biases for ro- bust visual question answering

    Jingliang Gu and Zhixin Li. Beyond language bias: Over- coming multimodal shortcut and distribution biases for ro- bust visual question answering. InProceedings of the 33rd ACM International Conference on Information and Knowl- edge Management, pages 3767–3771, 2024. 5, 16

    2024

  23. [31]

    Open-vocabulary object detection via vision and language knowledge distillation.arXiv preprint arXiv:2104.13921,

    Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation.arXiv preprint arXiv:2104.13921,

    arXiv

  24. [32]

    Many- modalqa: Modality disambiguation and qa over diverse in- puts

    Darryl Hannan, Akshay Jain, and Mohit Bansal. Many- modalqa: Modality disambiguation and qa over diverse in- puts. InProceedings of the AAAI Conference on Artificial Intelligence, pages 7879–7886, 2020. 3

    2020

  25. [33]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. InInternational conference on machine learning, pages 4904–4916. PMLR,

  26. [34]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. InProceedings of the IEEE conference on computer vision and pattern recognition...

    2017

  27. [35]

    Prompting visual-language models for efficient video understanding

    Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. InEuropean Conference on Computer Vi- sion, pages 105–124. Springer, 2022. 1

    2022

  28. [36]

    Fig- ureqa: An annotated figure dataset for visual reasoning

    Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkin- son, ´Akos K´ad´ar, Adam Trischler, and Yoshua Bengio. Fig- ureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300, 2017. 3

    2017 arXiv

  29. [37]

    Mdetr- modulated detection for end-to-end multi-modal understand- ing

    Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr- modulated detection for end-to-end multi-modal understand- ing. InProceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 1780–1790, 2021. 2

    2021

  30. [38]

    From word embeddings to document distances

    Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Wein- berger. From word embeddings to document distances. In International conference on machine learning, pages 957–

  31. [39]

    Clipstyler: Image style transfer with a single text condition

    Gihyun Kwon and Jong Chul Ye. Clipstyler: Image style transfer with a single text condition. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18062–18071, 2022. 1

    2022

  32. [40]

    Language-driven semantic seg- mentation

    Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic seg- mentation. InInternational Conference on Learning Rep- resentations, 2022. 2

    2022

  33. [41]

    Weinberger, Serge J

    Boyi Li, Kilian Q. Weinberger, Serge J. Belongie, Vladlen Koltun, and Ren ´e Ranftl. Language-driven semantic seg- mentation.CoRR, abs/2201.03546, 2022. 1

    2022 arXiv

  34. [42]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InInterna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 1, 2, 5, 6, 17

    2022

  35. [43]

    Visualbert: A simple and perfor- mant baseline for vision and language.arXiv preprint arXiv:1908.03557, 2019

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and perfor- mant baseline for vision and language.arXiv preprint arXiv:1908.03557, 2019. 2

    1908 arXiv

  36. [44]

    Oscar: Object-semantics aligned pre-training for vision-language tasks

    Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, ...

    2020

  37. [45]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  38. [46]

    Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering

    Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne. Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering. Advances in Neural Information Processing Systems, 36,

  39. [47]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 1, 2, 5, 17

    2023

  40. [48]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017. 17

    2017 arXiv

  41. [49]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.Advances in neural information processing systems, 32, 2019

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.Advances in neural information processing systems, 32, 2019. 2

    2019

  42. [50]

    Data- efficient image captioning of fine art paintings via virtual- real semantic alignment training.Neurocomputing, 490:163– 180, 2022

    Yue Lu, Chao Guo, Xingyuan Dai, and Fei-Yue Wang. Data- efficient image captioning of fine art paintings via virtual- real semantic alignment training.Neurocomputing, 490:163– 180, 2022. 3

    2022

  43. [51]

    Class-agnostic object detection with multi- modal transformer

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fa- had Shahbaz Khan, Rao Muhammad Anwer, and Ming- Hsuan Yang. Class-agnostic object detection with multi- modal transformer. InEuropean Conference on Computer Vision, pages 512–531. Springer, 2022. 2

    2022

  44. [52]

    Fine-grained visual classi- fication of aircraft.arXiv preprint arXiv:1306.5151, 2013

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classi- fication of aircraft.arXiv preprint arXiv:1306.5151, 2013. 3

    2013 arXiv

  45. [53]

    A multi-world ap- proach to question answering about real-world scenes based on uncertain input.Advances in neural information process- ing systems, 27, 2014

    Mateusz Malinowski and Mario Fritz. A multi-world ap- proach to question answering about real-world scenes based on uncertain input.Advances in neural information process- ing systems, 27, 2014. 3

    2014

  46. [54]

    Ask your neurons: A neural-based approach to answering questions about images

    Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz. Ask your neurons: A neural-based approach to answering questions about images. InProceedings of the IEEE interna- tional conference on computer vision, pages 1–9, 2015. 1

    2015

  47. [55]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. InProceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. 3

    2019

  48. [56]

    Taylor & Francis, 2008

    Paul F Marty and Katherine Burton Jones.Museum informat- ics: People, information, and technology in museums. Taylor & Francis, 2008. 1

    2008

  49. [57]

    Foundation model is efficient multimodal multitask model selector.arXiv preprint arXiv:2308.06262, 2023

    Fanqing Meng, Wenqi Shao, Zhanglin Peng, Chonghe Jiang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Foundation model is efficient multimodal multitask model selector.arXiv preprint arXiv:2308.06262, 2023. 1

    2023 arXiv

  50. [58]

    The rijksmuseum challenge: Museum-centered visual recognition

    Thomas Mensink and Jan Van Gemert. The rijksmuseum challenge: Museum-centered visual recognition. InProceed- ings of international conference on multimedia retrieval, pages 451–454, 2014. 3

    2014

  51. [59]

    Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories

    Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, Andr ´e Araujo, and Vittorio Ferrari. Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories. In Proceedings of the IEEE/CVF International Confer...

    2023

  52. [60]

    A dataset and a con- volutional model for iconography classification in paintings

    Federico Milani and Piero Fraternali. A dataset and a con- volutional model for iconography classification in paintings. Journal on Computing and Cultural Heritage (JOCCH), 14 (4):1–18, 2021. 3

    2021

  53. [61]

    Expanding language-image pretrained models for gen- eral video recognition

    Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for gen- eral video recognition. InEuropean Conference on Com- puter Vision, pages 1–18. Springer, 2022. 1

    2022

  54. [62]

    A survey of geospatial semantic web for cultural heritage

    Ikrom Nishanbaev, Erik Champion, and David A McMeekin. A survey of geospatial semantic web for cultural heritage. Heritage, 2(2):1471–1498, 2019. 1

    2019

  55. [63]

    Suppressing biased samples for robust vqa.IEEE Transactions on Multimedia, 24:3405– 3415, 2021

    Ninglin Ouyang, Qingbao Huang, Pijian Li, Yi Cai, Bin Liu, Ho-fung Leung, and Qing Li. Suppressing biased samples for robust vqa.IEEE Transactions on Multimedia, 24:3405– 3415, 2021. 5, 16

    2021

  56. [64]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311– 318, Philadelphia, Pennsylvania, USA, 2002. Associ...

    2002

  57. [65]

    Combined scal- ing for zero-shot transfer learning.Neurocomputing, 555: 126658, 2023

    Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scal- ing for zero-shot transfer learning.Neurocomputing, 555: 126658, 2023. 1

    2023

  58. [66]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  59. [67]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. InInternational confer- ence on machine learning, pages 8821–8831. Pmlr, 2021. 1

    2021

  60. [68]

    Fine-tuned clip models are efficient video learners

    Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tuned clip models are efficient video learners. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6545–6554, 2023. 1

    2023

  61. [69]

    Stylebabel: Artistic style tag- ging and captioning

    Dan Ruta, Andrew Gilbert, Pranav Aggarwal, Naveen Marri, Ajinkya Kale, Jo Briggs, Chris Speed, Hailin Jin, Baldo Fai- eta, Alex Filipkowski, et al. Stylebabel: Artistic style tag- ging and captioning. InEuropean Conference on Computer Vision, pages 219–236. Springer, 2022. 1, 3

    2022

  62. [70]

    Viske: Visual knowledge extraction and question answering by visual verification of relation phrases

    Fereshteh Sadeghi, Santosh K Kumar Divvala, and Ali Farhadi. Viske: Visual knowledge extraction and question answering by visual verification of relation phrases. InPro- ceedings of the IEEE conference on computer vision and pat- tern recognition, pages 1456–1464, 2015. 1

    2015

  63. [71]

    A dataset for multimodal question answering in the cultural heritage domain

    Shurong Sheng, Luc Van Gool, and Marie-Francine Moens. A dataset for multimodal question answering in the cultural heritage domain. InProceedings of the COLING 2016 Work- shop on Language Technology Resources and Tools for Dig- ital Humanities (LT4DH), pages 10–17. ACL, 2016. 2, 3

    2016

  64. [72]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 1

    2019

  65. [73]

    Bioclip: A vision foundation model for the tree of life

    Samuel Stevens, Jiaman Wu, Matthew J Thompson, Eliza- beth G Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M Dahdul, Charles Stewart, Tanya Berger- Wolf, et al. Bioclip: A vision foundation model for the tree of life. InProceedings of the IEEE/CVF Conference ...

  66. [74]

    Omniart: a large- scale artistic benchmark.ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 14 (4):1–21, 2018

    Gjorgji Strezoski and Marcel Worring. Omniart: a large- scale artistic benchmark.ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 14 (4):1–21, 2018. 1, 3

    2018

  67. [75]

    Multimodalqa: Complex question answering over text, tables and images.arXiv preprint arXiv:2104.06039, 2021

    Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. Multimodalqa: Complex question answering over text, tables and images.arXiv preprint arXiv:2104.06039, 2021. 3

    2021 arXiv

  68. [76]

    Ceci n’est pas une pipe: A deep convo- lutional network for fine-art paintings classification

    Wei Ren Tan, Chee Seng Chan, Hern ´an E Aguirre, and Kiyoshi Tanaka. Ceci n’est pas une pipe: A deep convo- lutional network for fine-art paintings classification. In2016 IEEE international conference on image processing (ICIP), pages 3703–3707. IEEE, 2016. 3

    2016

  69. [77]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023. 1

    2023 arXiv

  70. [78]

    The caltech-ucsd birds-200–2011 dataset

    C Wah, S Branson, P Welinder, P Perona, and S Belongie. The caltech-ucsd birds-200–2011 dataset. technical report california institute of technology.Technical re-port Califor- nia Institute of Technology, 2011. 3

    2011

  71. [79]

    Actionclip: A new paradigm for video action recognition.arXiv preprint arXiv:2109.08472, 2021

    Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition.arXiv preprint arXiv:2109.08472, 2021. 1

    2021 arXiv

  72. [80]

    Explicit knowledge-based rea- soning for visual question answering.arXiv preprint arXiv:1511.02570, 2015

    Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hen- gel, and Anthony Dick. Explicit knowledge-based rea- soning for visual question answering.arXiv preprint arXiv:1511.02570, 2015. 3

    2015 arXiv

  73. [81]

    Fvqa: Fact-based visual question an- swering.IEEE transactions on pattern analysis and machine intelligence, 40(10):2413–2427, 2017

    Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and An- ton Van Den Hengel. Fvqa: Fact-based visual question an- swering.IEEE transactions on pattern analysis and machine intelligence, 40(10):2413–2427, 2017. 3

    2017

  74. [82]

    Medclip: Contrastive learning from unpaired medical images and text.arXiv preprint arXiv:2210.10163, 2022

    Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text.arXiv preprint arXiv:2210.10163, 2022. 1

    2022 arXiv

  75. [83]

    Im- proving clip fine-tuning performance

    Yixuan Wei, Han Hu, Zhenda Xie, Ze Liu, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Im- proving clip fine-tuning performance. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 5439–5449, 2023. 1

    2023

  76. [84]

    Bam! the behance artistic media dataset for recognition beyond photography

    Michael J Wilber, Chen Fang, Hailin Jin, Aaron Hertzmann, John Collomosse, and Serge Belongie. Bam! the behance artistic media dataset for recognition beyond photography. InProceedings of the IEEE international conference on com- puter vision, pages 1202–1211, 2017. 1, 3

    2017

  77. [85]

    Ask me anything: Free-form vi- sual question answering based on knowledge from external sources

    Qi Wu, Peng Wang, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. Ask me anything: Free-form vi- sual question answering based on knowledge from external sources. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 4622–4630, 2016. 3

    2016

  78. [86]

    Language bias in visual question answering: A survey and taxonomy.arXiv preprint arXiv:2111.08531,

    Desen Yuan. Language bias in visual question answering: A survey and taxonomy.arXiv preprint arXiv:2111.08531,

    arXiv

  79. [87]

    Lit: Zero-shot transfer with locked-image text tuning

    Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 18123–18133, 2022. 1

    2022

  80. [88]

    The imet collection 2019 challenge dataset.arXiv preprint arXiv:1906.00901, 2019

    Chenyang Zhang, Christine Kaeser-Chen, Grace Vesom, Jennie Choi, Maria Kessler, and Serge Belongie. The imet collection 2019 challenge dataset.arXiv preprint arXiv:1906.00901, 2019. 2

    2019 arXiv

  81. [89]

    Tip-adapter: Training-free clip-adapter for better vision- language modeling.arXiv preprint arXiv:2111.03930, 2021

    Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision- language modeling.arXiv preprint arXiv:2111.03930, 2021. 2

    2021 arXiv

  82. [90]

    Extract free dense labels from clip

    Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. InEuropean Conference on Com- puter Vision, pages 696–712. Springer, 2022. 1, 2

    2022

  83. [91]

    Conditional prompt learning for vision-language mod- els

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language mod- els. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 16816–16825,

  84. [92]

    Learning to prompt for vision-language models.In- ternational Journal of Computer Vision, 130(9):2337–2348,

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models.In- ternational Journal of Computer Vision, 130(9):2337–2348,

  85. [93]

    Detecting twenty-thousand classes using image-level supervision

    Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Kr¨ahenb¨uhl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. InEuropean Confer- ence on Computer Vision, pages 350–368. Springer, 2022. 1, 2

    2022

  86. [94]

    Visual7w: Grounded question answering in images

    Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. InPro- ceedings of the IEEE conference on computer vision and pat- tern recognition, pages 4995–5004, 2016. 1

    2016

  87. [95]

    Who is the artist of the object?

    Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Llava-ϕ: Efficient multi-modal assistant with small language model.arXiv preprint arXiv:2401.02330, 2024. 2 A. Index Section Section Name 1 Index 2 Data 2.1 Data format 2.2 Example of instance 2.3 Data s...

    2024 arXiv

  88. [96]

    • These aggregators provide access to extensive digitized collections from major museums across Europe and America and offer structured data through platform- specific APIs

    Museum selection: • The dataset comprises 65 million data points, with 95% sourced from three major cultural aggregators: Digital Public Library of America - DPLA (24M), Europeana (20M), and the Smithsonian Institution (3.5M). • These aggregators provide access to extensive di...

  89. [97]

    Curation in- volved minimal edits: removing redundant attributes (in- ventory numbers, bibliographic info); extraneous symbols and numbers

    Data Cleaning:Each museum’s data was curated by a single domain expert to ensure consistency. Curation in- volved minimal edits: removing redundant attributes (in- ventory numbers, bibliographic info); extraneous symbols and numbers. Given high quality of museum data, focus wa...

  90. [98]

    Attribute-Value Structuring:While some museums pro- vided such structured data, others required parsing complete strings, with experts identifying logical separators and at- tribute boundaries through example-driven consensus

  91. [99]

    Which primary material is the object made of?

    Question Crafting: • To structure the attribute–value data for the visual ques- tion answering (VQA) task, we aligned our approach with natural human curiosity—formulating questions and ex- pecting concise answers—toward our goal of real-time deployment in interactive museum e...

  92. [100]

    For each object, we now have a list of images and a set of question-answer pairs, omitting the answers for which the value is not known

    Final Assembly:We download all the images from the collected image-urls. For each object, we now have a list of images and a set of question-answer pairs, omitting the answers for which the value is not known. Finally, for each museum we create 3 columns - image (having the li...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.