{"id":"975a93cb-02fe-45bf-95cb-eb97be95b89c","arxiv_id":"2412.01370","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new 65M-image, 200M-QA dataset for museum exhibits lets fine-tuned vision-language models beat general-purpose VLMs on museum attribute questions, especially on questions requiring background knowledge.","lead":"Researchers built MUSEUM-65, a collection of 65 million museum exhibit images paired with 200 million question-answer sets, and used it to teach two vision-language AI models how to answer visitor-style questions about artifacts. They report that museum-specific training dramatically improves accuracy over general-purpose models, with the larger LLaVA model handling history-heavy questions better than the smaller BLIP model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The main benchmark is in-distribution by construction: train and test share the same 63 question templates and metadata pipeline, so reported gains and SOTA comparisons may reflect template/format memorization rather than museum understanding.","rationale":"The reader's weakest_assumption and my analysis point to the same load-bearing issue: the test set is generated by the same template-and-metadata pipeline as the training set, so the benchmark cannot cleanly separate genuine museum understanding from template-to-answer memorization. This concern is concrete and localized: Sec. 3.2 explicitly describes manually crafting 63 questions from attribute-value pairs, and App. B.6 lists all 63 templates used for both training and evaluation. The paper does provide some mitigating evidence, notably the MUZE cross-evaluation (Table 17), where a model trained on MUSEUM-65 outperforms MUZE-trained BLIP on MUZE data, and the manually curated Visually Unanswerable set (Task 4), which goes beyond the attribute templates. These results suggest real transfer and prevent the concern from being fatal. However, they are small relative to the 6M-image test split, and the main accuracy numbers, especially the SOTA comparison restricted to Title and Place, could still be largely driven by format alignment with the 63 templates. The concrete test I propose would directly measure sensitivity to question phrasing while holding images and answers fixed. If the paraphrase holdout shows a small drop, the conditional verdict can be upgraded; if it shows a large drop, the headline claims need to be substantially softened. Since the reader already assigned CONDITIONAL with the same concern identified, my stress-test does not change the recommended verdict.","tokens_in":29655,"tokens_out":5479,"duration_ms":53357,"concrete_test":"Build a paraphrase holdout by having annotators rewrite each of the 63 templates into 5-10 semantically equivalent but lexically distinct phrasings (e.g., 'What material is the object made of?' becomes 'What substance was used to create this object?'), then apply these paraphrases to a fixed sample of 1,000 test objects with the same ground-truth answers. Evaluate LLaVA20mn-1ep, BLIP20mn-5ep, and two zero-shot SOTA models on both the original and paraphrased questions using identical scoring. If fine-tuned partial or complete precision drops by more than 15 points on paraphrases while zero-shot performance is roughly stable, the in-distribution concern is confirmed; if the drop is small, the main benchmark is more trustworthy than the template-overlap worry suggests.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the 6M-image test split measures genuine museum VQA ability. As constructed, it does not: Sec. 3.2 and App. B.6 show that every train and test question is generated from the same 63 human-written templates applied to the same attribute-value metadata, and Sec. 3.5 provides no evidence that the test distribution differs from the training distribution or that an object-level split was enforced. Fine-tuned models are therefore rewarded for learning the template-to-metadata mapping, including exact output formatting, rather than for understanding exhibits from images. The headline SOTA comparison in Table 1 and the General VQA results in Table 18 are consistent with this concern: zero-shot VLMs produce varied phrasings that exact-match metrics penalize, while fine-tuned models output the exact metadata strings they were trained to reproduce. The manually constructed Visually Unanswerable set (515 test pairs, Task 4) and the MUZE cross-evaluation in Table 17 provide genuinely partial out-of-distribution evidence, so the result is not likely fabricated; however, the magnitude of the reported gains and the 'reasoning' interpretation are not supported by the main benchmark as it stands.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces MUSEUM-65, a large-scale multilingual dataset of museum exhibit images paired with question-answer pairs generated from museum metadata, and fine-tunes BLIP and LLaVA models on subsets of it. The authors propose five VQA tasks: general VQA, category-wise VQA, multi-angle VQA, visually unanswerable questions, and multilingual VQA, and report that fine-tuned models outperform zero-shot state-of-the-art models on attribute questions, with the fine-tuned LLaVA model performing best. They also report a cross-evaluation on MUZE, a comparison with human experts, and an ablation on place of origin.","tokens_in":29793,"tokens_out":6565,"duration_ms":54317,"significance":"The paper's main potential contribution is the scale and public availability of the dataset, and the MUZE cross-evaluation in Table 17 provides a useful external check that training on MUSEUM-65 transfers to another benchmark. The manually constructed Visually Unanswerable set is a genuine attempt to go beyond purely visual grounding. However, the central benchmarking claim is weakened by the in-distribution construction of the test set, and the 'reasoning' claim rests on a small test set with modest partial-precision gains. The dataset could still be a valuable resource if the authors restructure the evaluation to include out-of-distribution generalization checks and temper the claims accordingly.","major_comments":[{"comment":"The main benchmark is in-distribution by construction: both training and test questions are generated from the same 63 hand-written templates applied to the same attribute-value metadata, and Sec. 3.5 does not describe any object-level or template-level split. Consequently, the gains reported in Table 1 and Table 18 may largely reflect learning the template-to-metadata mapping and the exact output formatting rather than understanding museum exhibits from images. Please enforce object-level and template-level separation (e.g., hold out entire question templates, or at least verify that no object appears in both train and test), and report results on such splits, or restrict the headline claims to the manually curated Task 4 and the MUZE cross-evaluation in Table 17.","section":"Sec. 3.2, 3.5, App. B.6 (Table 12)"},{"comment":"The definition of 'Complete Precision' in Sec. 4.1 is the fraction of questions with precision=1.0, which only requires that every word in the prediction appears in at least one valid answer; it does not require the prediction to cover the full ground-truth answer. Under this metric, a short correct substring scores 100%, so the comparison between zero-shot (verbose) and fine-tuned (terse) models is inflated by output length and phrasing differences. Please report exact-match accuracy and recall-based metrics alongside, or justify why the defined metric is appropriate for the cross-model comparison.","section":"Sec. 4.1, Table 18"},{"comment":"The claim that large VLMs 'excel in queries requiring deeper historical context and reasoning' (abstract) rests on 515 test pairs in Task 4. On these pairs, LLaVA20mn-1ep reaches 29.7% partial precision versus 27.18% for LLaVA without fine-tuning and 25.83% complete precision versus 1.55%, with no confidence intervals or significance tests. The partial-precision gap is small, and many of these questions ask for world knowledge about the object's country of origin, an attribute on which the model was fine-tuned. Please provide bootstrap confidence intervals, a breakdown by question type, and an analysis of failure cases to substantiate the reasoning interpretation.","section":"Sec. 5.5, Tables 5 and 6"},{"comment":"The statement that fine-tuned models 'outperform human experts across all categories' is based on 850 QA pairs answered by 10 experts and scored against museum metadata strings with the same lexical metrics. Humans were not given the metadata source, and the metric rewards exact string overlap, so this comparison conflates metadata retrieval with domain expertise. The IAA paragraph reports 52.7% agreement and a 6.25% chance baseline, but the number of categories and the simulation procedure are not described. Please provide the full protocol or soften the claim.","section":"Sec. 5.3, Fig. 6"},{"comment":"The abstract's statement that 'the complete dataset is labeled by museum experts' is not supported by the curation description in Sec. 3.2, where experts clean existing museum metadata and manually write 63 question templates; the per-item answers are museum metadata values, not expert-authored labels. Please revise the wording to describe the annotation process accurately.","section":"Abstract and Sec. 3.2"}],"minor_comments":[{"comment":"The model names 'LLaVa nofinetune' and 'LLaVa-ours' are inconsistently capitalized; please use one consistent name (e.g., LLaVA) throughout.","section":"Table 1"},{"comment":"The rows 'LLaV A 20mm-1ep' and 'BLIP 20mm-5ep' contain the typo '20mm'; these should be '20mn'.","section":"Tables 5 and 6"},{"comment":"The caption refers to 'LLaV A20mn-5ep', but the main text's best model is LLaVA20mn-1ep; please clarify which model produced the examples.","section":"Fig. 10 caption"},{"comment":"The sentence 'The links to the curated and raw datasets can be found here: MUSEUM-65' does not display an actual URL; please include the full link in the manuscript.","section":"App. B.4"},{"comment":"The metric names 'Complete Precision' and 'Partial Precision' are misleading because they are not exact-match fractions; consider renaming them to something like 'No-False-Positive Precision' and 'Any-Hit Precision' to avoid confusion.","section":"Sec. 4.1"},{"comment":"The section heading 'Datasest Curation Process' contains a typo; it should be 'Dataset Curation Process'.","section":"App. B.5"},{"comment":"The claim that 'benefits distribute evenly' across continents is supported only by point estimates in Table 8; no confidence intervals or significance tests are provided, so the strength of this claim is overstated.","section":"Sec. 3.3, Table 8"}],"recommendation":"major_revision","confidential_remarks":"This is a dataset paper, and the dataset release is central to its value. Given the in-distribution evaluation issues, the paper would be substantially strengthened by releasing a leaderboard with held-out question templates and object-level splits, and by making the Visually Unanswerable set the primary evidence for the reasoning claim. I also note that reference [5] (MUZE) is from the same group; the comparison in Table 17 should ensure that training and evaluation protocols are identical and that the result is not due to different preprocessing of the MUZE data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — This is a potentially valuable dataset paper with an evaluation section that overreaches. The resource is the story: MUSEUM-65, 65M images and 200M QA pairs scraped from aggregators and museums, more than two orders of magnitude larger than MUZE or VISCOUNTH for cultural-heritage VQA. The five-task benchmark (general, category-wise, multi-angle, visually unanswerable, multilingual) is sensible, and the authors did real curation work: experts cleaned attributes, wrote 63 question templates with varied phrasing, cross-checked quality, and report bias analyses. That part deserves credit.\n\nThe experiments are honestly described and mostly reproducible in principle: fine-tuning BLIP and LLaVA on increasingly large subsets consistently improves exact-match and semantic metrics, and the cross-evaluation on MUZE in Table 17 is the strongest external check in the paper. The multi-angle robustness result is clean. I would trust the main empirical observation that fine-tuning on this scale helps.\n\nThe soft spots are the claims built on top of that observation. The main benchmark is in-distribution by construction: train and test share the same 63 question templates and the same attribute-value metadata pipeline (Sec 3.2, App B.6), so the large gains partly measure learning the template-to-answer mapping, not museum understanding. The headline SOTA comparison in Table 1 is restricted to two attribute types (title, place), and the abstract's \"excel in queries requiring deeper historical context and reasoning\" rests on 515 test pairs where the best model gets 25-31% partial precision. Those are not fabricated results—the manually built Visually Unanswerable set and MUZE cross-eval give genuine out-of-distribution signal—but the claims are stronger than the evidence.\n\nThere is also a practical blocker: the dataset and code are promised but no link is given. For a paper whose contribution is the dataset, that matters. If the release materializes, this becomes a solid domain-specific resource; as submitted, it is a promising but unevaluable dataset with an overclaimed benchmark.\n\nWho is it for: people working on cultural-heritage VQA, domain-specific VQA, or VLM fine-tuning at scale. It deserves a serious referee, conditional on the authors actually releasing the data and either adding an external open-ended evaluation or toning down the reasoning/SOTA claims. I would send it to review with those conditions front and center.","headline":"A genuinely large and well-curated museum VQA dataset with an overclaimed in-distribution evaluation; worth refereeing if the promised release actually happens.","tokens_in":30517,"tokens_out":1783,"would_cite":false,"duration_ms":14799,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 65M-image, 200M-question museum dataset lifts fine-tuned vision-language models far above zero-shot generalists on exhibit question answering.","keywords":["visual question answering","vision-language models","museum exhibits","cultural heritage","fine-tuning","multimodal dataset","multilingual VQA","multi-angle evaluation"],"falsifier":"Evaluate the fine-tuned models on a test set of questions written by museum visitors or domain experts that share no templates with the 63 training questions; if performance falls to near the zero-shot baseline while in-distribution scores stay high, the reported gains are template memorization rather than museum understanding.","tokens_in":29361,"feed_emoji":"🏛️","tokens_out":5022,"duration_ms":45040,"temperature":0.7,"pith_summary":"This paper argues that museum exhibits can be understood by vision-language models if the models are fine-tuned on a sufficiently large, expert-curated dataset. It introduces MUSEUM-65, a collection of 65 million exhibit images paired with 200 million question-answer pairs, and trains two model families on subsets of it. Across five visual-question-answering tasks, the fine-tuned models substantially outperform zero-shot state-of-the-art VLMs on attribute questions such as title, place of origin, and material. The paper also reports that LLaVA, the instruction-tuned model with a larger language backbone, outperforms BLIP on questions requiring historical context and reasoning, while both models handle visually grounded questions well. If correct, this establishes domain-specific large-scale fine-tuning as a practical route to museum AI applications.","feed_headline":"65M museum images lift VLMs past zero-shot SOTA on exhibit QA","feed_subtitle":"A new 200M-pair dataset pushes fine-tuned models to 80 percent partial precision on exhibit questions.","key_machinery":"The load-bearing mechanism is MUSEUM-65 itself: museum metadata scraped from three international aggregators and twelve additional museums is cleaned into attribute-value pairs, then 63 manually crafted question templates turn each pair into a QA instance with the attribute value as answer. Fine-tuning BLIP, an encoder-decoder VLM with vision-language aligned embeddings, and LLaVA, an instruction-tuned large language model with vision-language reasoning, on subsets of 1, 10, or 20 million samples supplies the comparison. The five-task benchmark, covering general VQA, category-wise VQA, multi-angle robustness, visually unanswerable questions, and multilingual queries, is what converts raw accuracy into claims about viewpoint robustness, reasoning, and language transfer.","core_discovery":"The central claim is that a large, structured, expert-labeled dataset of museum metadata, reformulated into natural-language questions, is sufficient to make existing vision-language models perform at a level far above zero-shot generalists on exhibit understanding. The evidence: BLIP fine-tuned on 20 million samples reaches 71.51% partial precision and 60.58% complete precision on general VQA, while LLaVA fine-tuned on 20 million samples for one epoch reaches 81.25% partial precision and 63.96% complete precision, with zero-shot SOTA models scoring in the low 20s to 40s on the same attribute questions. On the manually built Visually Unanswerable set, LLaVA reaches 29.7% partial precision on unseen images while BLIP drops to 3.3%. The enabling factor is the dataset's attribute-value structure: museum metadata is converted by human-written templates into 200 million QA pairs, giving models repeated exposure to the exact vocabulary and reasoning patterns of exhibit catalogues.","pith_inferences":["A natural next check the paper leaves implicit is whether the gains survive on naturally occurring visitor questions whose phrasing was not drawn from the 63 templates; if they do not, the reported scores would mostly reflect template memorization.","The multilingual results suggest a concrete extension: fine-tuning on the 15 million non-English samples in MUSEUM-65 could reverse the language forgetting observed in the English-only LLaVA model.","Because the dataset releases raw metadata alongside the curated QA pairs, it could support retrieval-based and knowledge-augmented VQA systems that answer visually unanswerable questions with external references.","The attribute-value structure also invites a diagnostic use beyond training: probing which attribute categories remain hard after fine-tuning, such as title and subject, could guide where museum metadata needs to be richer."],"forward_implications":["Fine-tuned VLMs can support museum visitor question answering for attribute-level queries such as title, material, and place of origin at much higher accuracy than general-purpose zero-shot models.","Instruction-tuned, LLM-based VLMs are the more promising architecture when questions require historical context or reasoning beyond what is visible in the image.","Domain-specific large-scale fine-tuning is necessary for strong performance on nuanced museum attributes, since zero-shot SOTA models lag far behind the fine-tuned models.","The dataset and benchmark provide a reusable testbed for five VQA tasks, including multilingual queries and viewpoint changes, which are directly relevant to real museum deployments.","Cross-evaluation on the MUZE benchmark suggests that models trained on MUSEUM-65 can transfer to existing cultural-heritage datasets and outperform models trained on those datasets themselves."],"supporting_citations":[{"why":"Supplies the BLIP model architecture and fine-tuning protocol used as the smaller vision-language baseline.","marker":"[42]"},{"why":"Supplies the LLaVA instruction-tuned model and LoRA fine-tuning protocol used as the large vision-language baseline.","marker":"[47]"},{"why":"Provides the MUZE dataset used for cross-evaluation and for the benchmark comparison showing MUSEUM-65-trained models outperform MUZE-trained models.","marker":"[5]"},{"why":"Provides the prior large-scale cultural-heritage VQA dataset VISCOUNTH, which the paper compares against in scale and domain coverage.","marker":"[8]"},{"why":"Defines the BLEU n-gram metric used to evaluate answer quality.","marker":"[64]"},{"why":"Defines Word Mover's Distance, used for the semantic top-1 accuracy metric.","marker":"[38]"},{"why":"Supplies the CLIP visual backbone inherited by both BLIP and LLaVA in this paper.","marker":"[66]"}],"fun_headline_variants":["65M museum images fine-tune VLMs to 81% exhibit QA precision","Expert-labeled museum QA: 200M pairs lift VLMs to 81% on exhibit questions","Fine-tuned LLaVA beats zero-shot SOTA on museum exhibit VQA by 2x","Museum QA dataset: 200M pairs turn VLMs into exhibit experts","From zero-shot to 81%: museum exhibit VQA with 65M images"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The held-out test questions are generated by the same 63 templates and the same metadata pipeline as the training data, so the central claim assumes that high scores on this in-distribution test reflect genuine museum understanding rather than memorized template-answer patterns.","fun_headline_variants_meta":{"raw":{"variants":["65M museum images fine-tune VLMs to 81% exhibit QA precision","Expert-labeled museum QA: 200M pairs lift VLMs to 81% on exhibit questions","Fine-tuned LLaVA beats zero-shot SOTA on museum exhibit VQA by 2x","Museum QA dataset: 200M pairs turn VLMs into exhibit experts","From zero-shot to 81%: museum exhibit VQA with 65M images"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000524,"raw_usage":{"total_tokens":2580,"prompt_tokens":1041,"completion_tokens":1539,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":1426}},"tokens_in":657,"tokens_out":1539,"duration_ms":10882,"temperature":1.0,"reasoning_tokens":1426,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:24:42.115107+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the fine-tuned models on a test set of questions written by museum visitors or domain experts that share no templates with the 63 training questions; if performance falls to near the zero-shot baseline while in-distribution scores stay high, the reported gains are template memorization rather than museum understanding.","supporting_citations":[{"cited_title":"Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation","cited_arxiv_id":null,"evidence_quote":"Supplies the BLIP model architecture and fine-tuning protocol used as the smaller vision-language baseline."},{"cited_title":"Visual instruction tuning, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the LLaVA instruction-tuned model and LoRA fine-tuning protocol used as the large vision-language baseline."},{"cited_title":"Taming CLIP for Fine-grained and Structured Visual Understanding of Museum Exhibits","cited_arxiv_id":"2409.01690","evidence_quote":"Provides the MUZE dataset used for cross-evaluation and for the benchmark comparison showing MUSEUM-65-trained models outperform MUZE-trained models."},{"cited_title":"Bleu: a method for automatic evaluation of machine translation","cited_arxiv_id":null,"evidence_quote":"Defines the BLEU n-gram metric used to evaluate answer quality."},{"cited_title":"From word embeddings to document distances","cited_arxiv_id":null,"evidence_quote":"Defines Word Mover's Distance, used for the semantic top-1 accuracy metric."}],"review_version":1}