Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

PuzzleGPT: Emulating Human Puzzle-Solving Ability for Time and Location Prediction

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PuzzleGPT claims that time and location prediction from images is a five-skill puzzle, and that a zero-shot modular pipeline of frozen components solves it better than end-to-end vision-language models.

desk verdict A thoughtful modular pipeline whose TARA SOTA claim is not yet supported because the new metrics were not shown to be applied to baselines; WikiTilo is the stronger evidence. read the letter →

arxiv 2501.14210 v1 pith:HEUFK7IG submitted 2025-01-24 cs.CV cs.AIcs.LG

Hammad Ayyubi , Xuande Feng , Junzhang Liu , Xudong Lin , Zhecan Wang , Shih-Fu Chang This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords timeandlocationpredictionpuzzle-solvingmodularvision-languagepipelinezero-shotreasoninghierarchicalcombinationwebretrievalTARAdatasetWikiTilo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that predicting the time and place a photograph was taken is a multi-step puzzle that should be solved by decomposing it into separate skills rather than by asking a single vision-language model to answer in one pass. It introduces PuzzleGPT, a zero-shot pipeline of frozen components: a perceiver extracts visual clues, a reasoner turns each clue into time and location candidates, a combiner merges candidates hierarchically, a noise filter rejects dubious ones, and a web retriever supplies outside knowledge when local reasoning stalls. On the TARA and WikiTilo datasets the authors report state-of-the-art performance, beating large end-to-end vision-language models including GPT-4o and automatic pipeline generators such as VisProg, and rivaling or surpassing finetuned models. The payoff, if true, is that interpretable modular reasoning can outperform monolithic models on tasks that require combining perception, world knowledge, and common sense.

What carries the argument

The load-bearing object is the confidence-based hierarchical combiner. It reasons over entities at increasing granularity—individual clues, pairs, triplets, then all clues—and records each accepted candidate in hash maps for location and time; a candidate that reaches a hash threshold triggers early stopping, while the highest-vote candidate is the final answer if no threshold is reached. Around it sit a frozen vision-language perceiver, an LLM reasoner, a vision-language noise filter that asks whether the image could plausibly be taken at a candidate place or time, and a CLIP-scored web retrieval step that keeps only results above a retrieval threshold. The mechanism's job is to balance sparse information from single clues against noise from indiscriminate combination, and to decide when external search is needed.

What would settle it

Re-run BLIP-2, InstructBLIP, LLaVA, GPT-4o, and VisProg on the TARA test set and score them with the same GeoPy-standardized location labels and brevity-penalized X-F1-beta used for PuzzleGPT. If the resulting margins fall below the reported 32% and 38% improvements, or if any baseline surpasses PuzzleGPT on standardized location accuracy, the paper's central comparative claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that human puzzle-solving ability for time and location inference can be emulated by five abstract skills, each instantiated by a frozen pretrained module, and that this expert pipeline outperforms single-pass vision-language models and automatic code-generation pipelines. Time and location are predicted as ranked candidate sets accumulated in hash maps; the combiner traverses hierarchy levels—single clues, pairs, triplets, then all clues—and stops early when a candidate reaches a confidence threshold. The paper further claims that standardizing location labels with GeoPy and adding a brevity penalty to Example-F1 makes evaluation fairer, and that under these metrics PuzzleGPT achieves 22.99% standardized location accuracy and 43.72 time X-F1 on TARA, with state-of-the-art accuracy and F1 on WikiTilo's time and region choices. The method is intended as evidence that current vision-language models lack the ability to integrate multiple skills simultaneously.

Load-bearing premise

The state-of-the-art claim rests on the assumption that the newly introduced evaluation metrics—GeoPy-standardized location accuracy and brevity-penalized F1—are applied uniformly to PuzzleGPT and to every model it is compared with, since the headline margins use previously reported baseline numbers rather than baselines re-scored under the same rules.

Editorial extensions

If this is right

  • Task-specific modular pipelines can beat end-to-end vision-language models on multi-step visual reasoning without any training.
  • Hierarchical combination with early stopping beats both single-clue reasoning and indiscriminate all-clue fusion.
  • Web retrieval is a major contributor to time prediction, where local visual clues are often insufficient.
  • The success on WikiTilo indicates that the design transfers across datasets with different label structures.
  • The approach is interpretable: each final prediction can be traced to a clue or web snippet that produced it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the decomposition is right, the same five-module recipe should transfer to other evidence-combination tasks, such as dating archival photographs or reconstructing event timelines from social media posts.
  • The reliance on a proprietary GPT-based reasoner suggests the margins may shrink with open models; the paper's own LLaMA 3.1 ablation shows a large drop, so practical portability depends on replacing that module.
  • A conservative reading of the headline gains: the paper re-scores its own outputs with GeoPy-standardized labels and a brevity-penalized F1 while taking baseline numbers from prior reports, so the true margins will only be known once all systems are re-scored under identical rules.
  • The confidence threshold is a single hyperparameter controlling interpretability and noise; one testable extension is making the threshold adaptive per image or per clue type.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes PuzzleGPT, a modular zero-shot pipeline for time and location prediction from images, consisting of a perceiver, reasoner, combiner, noise filter, and web retriever, all built from frozen VLMs/LLMs. The authors introduce two new evaluation metrics for the TARA dataset — Standardized Accuracy (Std. Acc) and brevity-penalized Example-F1 (X-F1β) — and report state-of-the-art results on TARA and WikiTilo, claiming large margins over large VLMs such as BLIP-2, LLaVA, GPT-4o, and over finetuned classification models. The central claim is that a modular, interpretable, retrieval-augmented pipeline can outperform end-to-end VLMs on a complex multi-step reasoning task.

Significance. If the reported results are valid, the paper would make a meaningful contribution by demonstrating that a well-designed modular pipeline of frozen components can beat much larger end-to-end VLMs on a complex visual reasoning task, while also providing interpretability. The paper includes useful ablations of the main modules (hierarchical combination, noise filter, retriever) and reports component-wise analyses. The proposed evaluation metrics address a real problem with TARA's noisy location labels. However, the validity of the headline quantitative claims hinges on whether all baselines were scored under the same evaluation protocol, which the paper does not establish. As presented, the comparison is not apples-to-apples in Table 2, and this weakens the state-of-the-art claim.

major comments (4)
  1. [§4.1, Table 2] The headline claim that PuzzleGPT 'surpasses finetuned methods on location reasoning' rests on an invalid comparison. In Table 2, PuzzleGPT's Location accuracy is marked with an asterisk denoting Std. Acc (22.99*), while the finetuned baselines CLIP, CLIP+, CLIP+Seg, and QR-CLIP are reported using the original raw Location Acc (15.72, 16.46, 19.31). Since Std. Acc forgives extra label fields that raw exact-match penalizes, the margin between 22.99* and 19.31 may be an artifact of the metric rather than of model skill. All rows must be scored under the same metric, or the table must present both raw and standardized accuracy for every method.
  2. [§4.1, Metric; Table 1] The paper does not document that the baseline predictions in Table 1 (BLIP-2, LLaVA, GPT-4o, VisProg, etc.) were re-scored with GeoPy-standardized labels and the X-F1β brevity penalty. The paragraph introducing the metrics says 'All our experiments and ablations on TARA use the more correct X-F1β and Location Std. Acc metric,' but it does not state that baselines' raw outputs were passed through the same standardization and brevity-penalty pipeline. Without this, the abstract's 'at least 32% and 38%' improvements could reflect the change in evaluation metric rather than model capability. Please provide the exact baseline evaluation protocol, ideally including the raw baseline outputs or a re-scored comparison table.
  3. [§3, Confidence-based Hash Thresholding; Figure 5] The two principal hyperparameters, Hash Threshold (HT) and Retrieval Threshold (RT), are selected through ablations run on the TARA dataset, but no validation split is described. Figure 5 shows performance peaking at HT=5 and RT=90, with the text saying 'The hash threshold HT is initially set to 5.' If these thresholds were chosen by inspecting test-set performance, the reported numbers include selection bias. The authors should either fix the thresholds before evaluation, use a separate validation split, or report the sensitivity of the final results to the choice of thresholds on held-out data.
  4. [§4.1, Results; Table 2; Abstract] The time-prediction claim is overstated. In Table 2, PuzzleGPT's Time Acc is 0.30, far below QR-CLIP's 3.53, and its Time X-F1 of 43.72 is below QR-CLIP's 47.89. The text acknowledges 'comparable performance on time prediction,' but the abstract and contributions imply state-of-the-art on TARA without qualification. The margins 'at least 32% and 38%' in the abstract are not supported for time prediction under any consistent reading of Table 2. Please clarify that the SOTA claim on TARA is limited to the zero-shot setting, or for the finetuned comparison, specify that time accuracy is not competitive.
minor comments (5)
  1. [Abstract] The abstract mentions 'GPT-4V' in one sentence and 'GPT-4o' elsewhere in the paper; the table uses GPT-4o. Please unify the model name.
  2. [§4.1, Metric] The sentence 'In addition to city, country, and country, some labels contain additional information' contains a duplicated word ('country').
  3. [§4.1, Table 2 caption] The footnote '* denotes Std. Acc' is attached only to the location accuracy cell for PuzzleGPT, but it should be clear whether the Location X-F1 column for PuzzleGPT (56.11) also uses standardized labels; if so, the corresponding baseline X-F1 values should also be standardized for a fair comparison.
  4. [§4.1, Ablation Studies; Figure 5] The paragraph 'Confidence thresholding matters in hierarchical' states 'the best performance is reached at threshold=90' without specifying that this refers to the Retrieval Threshold; the top row of Figure 5 is the Hash Threshold, and the text should name each threshold explicitly in each sentence.
  5. [§4.2, Table 6] The paper reports 'state-of-the-art performance on WikiTilo' and then immediately notes that Country Acc is slightly worse than the previous best. Consider phrasing the contribution as SOTA on Time and Region, with a clear qualification about Country accuracy.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: PuzzleGPT's pipeline is modular and its reported gains rest on external benchmarks; the metric-consistency and threshold-tuning concerns are evaluation-validity issues, not circularity.

full rationale

The paper's central claim is an empirical comparison against external baselines on TARA and WikiTilo, not a derivation that reduces to its own inputs. PuzzleGPT is explicitly built from frozen, pretrained components: "the Perceiver is a frozen VLM that is prompted with a respective query to extract each entity," "Reasoner in PuzzleGPT is an LLM," and the Combiner, Noise Filter, and Online Retriever are all described as modular pipeline stages. None of these modules is fitted to the target labels or to the evaluation metrics, so the reported accuracy and F1 numbers are not equal to any fitted parameter by construction. The comparison tables and ablations are external-checked against BLIP-2, InstructBLIP, LLaVA, GPT-4o, VisProg, ViperGPT, IdealGPT, and finetuned models, which means the SOTA claim is falsifiable rather than self-referential. Two concerns do appear in the paper but are not circularity in the sense of deriving X from Y by definition. First, the evaluation section states: "All our experiments and ablations on TARA use the more correct X-F1β and Location Std. Acc metric, except for Table 2 where we use X-F1 and Location Acc for direct comparison against previously reported results." Table 2 then marks PuzzleGPT's location accuracy with an asterisk denoting Std. Acc while baseline location accuracies are raw Acc, so the headline "at least 32% and 38%" margins depend on metrics being applied uniformly across rows. This is a comparison-validity or evaluation-hygiene issue, not a circular reduction: the paper does not define PuzzleGPT's accuracy in terms of the baselines' accuracy, nor does it fit a parameter that mechanically produces the margin. Second, the hash threshold and retrieval threshold are hyperparameters ("The hash threshold HT is initially set to 5"), and Figure 5 ablates them; if those thresholds were selected on the TARA test set, that would be test-set tuning that inflates the reported numbers, but it would still be an overfitting/leakage problem rather than a circular derivation. There are also self-citations to prior work by overlapping authors (e.g., Lin et al. 2023, Fu et al. 2024), but these are not load-bearing: they are background references on vision-language retrieval and visual reasoning, and no uniqueness theorem or fitted prior from those papers is invoked to force PuzzleGPT's design or results.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central performance claims rest on two fitted hyperparameters (HT and RT) and on an evaluation protocol that mixes standardized and raw metrics. There are no new physical or conceptual entities introduced. The strongest extra load is the unstated assumption that threshold tuning was done on a validation split and that the metric changes are neutral across methods.

free parameters (2)
  • Hash Threshold (HT) = 5
    Section 4.1 and Figure 5: The threshold at which a candidate is accepted and the pipeline stops is set to 5, and the ablation on TARA shows 5 is the best among 3, 5, and 7. The paper does not state whether this selection used a validation or test split.
  • Retrieval Threshold (RT) = 90
    Section 4.1 and Figure 5: Web retrieval results are kept only if their similarity score exceeds RT. The ablation on TARA shows 90 is best among 80, 85, 90, and 95, again without a stated validation split.
assumptions (3)
  • domain assumption The GeoPy-based standardization of TARA location labels preserves the relative ordering of methods and can be applied to model predictions in the same way it is applied to labels.
    Section 4.1, Metric paragraph: 'we standardize all locations into city, country, and continent using GeoPy.' The paper does not analyze whether the standardization changes the ranking of methods or whether baseline predictions were standardized at all.
  • ad hoc to paper The hyperparameters HT and RT were selected using a validation set, not the TARA test set.
    The ablations in Figure 5 are presented as tuning curves on TARA without any mention of a validation split. If the test set was used for selection, the reported numbers are partially fit to the test set.
  • domain assumption The frozen components BLIP-2, GPT-3.5, and CLIP provide sufficiently accurate perception, reasoning, and retrieval matching for the pipeline to work.
    Section 3: The perceiver, reasoner, and retrieval scorer are all pretrained frozen models. The paper provides only qualitative failure analysis and no quantitative error decomposition for these components.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PuzzleGPT: Emulating Human Puzzle-Solving Ability for Time and Location Prediction." pith.science (2026). https://pith.science/paper/HEUFK7IG

@misc{pith2026250114210,
  author       = {Pith},
  title        = {Pith review of: PuzzleGPT: Emulating Human Puzzle-Solving Ability for Time and Location Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HEUFK7IG}},
  note         = {Machine review of arXiv:2501.14210}
}
read the original abstract

The task of predicting time and location from images is challenging and requires complex human-like puzzle-solving ability over different clues. In this work, we formalize this ability into core skills and implement them using different modules in an expert pipeline called PuzzleGPT. PuzzleGPT consists of a perceiver to identify visual clues, a reasoner to deduce prediction candidates, a combiner to combinatorially combine information from different clues, a web retriever to get external knowledge if the task can't be solved locally, and a noise filter for robustness. This results in a zero-shot, interpretable, and robust approach that records state-of-the-art performance on two datasets -- TARA and WikiTilo. PuzzleGPT outperforms large VLMs such as BLIP-2, InstructBLIP, LLaVA, and even GPT-4V, as well as automatically generated reasoning pipelines like VisProg, by at least 32% and 38%, respectively. It even rivals or surpasses finetuned models.

Figures

Figures reproduced from arXiv: 2501.14210 by the authors.

Figure 1
Figure 1. An illustration of different skills (Puzzle-like) required for solving Time and Reasoning prediction tasks. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. PuzzleGPT overview depicting each of the components – Perceiver, Reasoner, Combiner, Noise Filter, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Extra location information in the label causes even correct prediction to be classified as wrong with exact-match accuracy metric. We mitigate this by label standardization. insufficient to reason complicated tasks based only on static knowledge priors obtained through per￾taining. From a human perspective as well, we try to access online resources once our knowledge is insufficient for a particular task. To mimic s… view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Top: Ablation on Hash Threshold (HT): peak [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 4
Figure 4. Figure 4: An illustration of the importance of confidence-based hierarchical combination of informa￾tion. 1st Hierarchy leads to a scarcity of information, while 3rd Hierarchy is noisy, and underscores the need for a confidence-based hierarchical combination. pipelines, contrast…
Figure 8
Figure 8. Figure 8: Frequency of retrieval and no retrieval be [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: We illustrate a positive case study for PuzzleGPT depicting the reasoning process it follows to arrive at [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Two samples for negative case studies. I thought there is a statue, and I also find that this is an event include monuments and lights! TEXT BLM are shown! Based on Statue: United States Based on TEXT: United States, 2020s Based on Monument: Not Sure Based on BLM and …
Figure 11
Figure 11. Figure 11: Another positive case study. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GETReason: Enhancing Image Context Extraction through Hierarchical Multi-Agent Reasoning

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A multi-agent vision-language framework that extracts event, time, and location from public event images, evaluated with a new soft metric on VLM-augmented datasets.

Reference graph

Works this paper leans on

32 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski,...

    arXiv 2022

  4. [4]

    Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 39--48

    work page 2016

  5. [5]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA : V isual Q uestion A nswering. In International Conference on Computer Vision (ICCV)

    work page 2015

  6. [6]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. https://arxiv.org/abs/2305.06500 Instructblip: Towards general-purpose vision-language models with instruction tuning . Preprint, arXiv:2305.06500

    arXiv 2023

  7. [7]

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404.12390

    arXiv 2024

  8. [8]

    Xingyu Fu, Ben Zhou, Ishaan Preetam Chandratreya, Carl Vondrick, and Dan Roth. 2022. https://arxiv.org/abs/2203.00758 There is a time and place for reasoning beyond the image . Preprint, arXiv:2203.00758

    work page Pith review arXiv 2022

Show all 32 references
  1. [9]

    Tanmay Gupta and Aniruddha Kembhavi. 2022. https://arxiv.org/abs/2211.11559 Visual programming: Compositional visual reasoning without training . Preprint, arXiv:2211.11559

    2022 arXiv

  2. [10]

    Xiaotian Han, Quanzeng You, Yongfei Liu, Wentao Chen, Huangjie Zheng, Khalil Mrini, Xudong Lin, Yiqi Wang, Bohan Zhai, Jianbo Yuan, et al. 2023. Infimm-eval: Complex open-ended reasoning evaluation for multi-modal large language models. arXiv e-prints, pages arXiv--2311

    2023

  3. [11]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. 2019. https://arxiv.org/abs/1902.09506 Gqa: A new dataset for real-world visual reasoning and compositional question answering . Preprint, arXiv:1902.09506

    2019 arXiv

  4. [12]

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern reco...

    2017

  5. [13]

    Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou, and Luo Si. 2022. https://arxiv.org/abs/2205.12005 mplug: Effective and efficient vision-language learning by cro...

    2022 arXiv

  6. [14]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. https://arxiv.org/abs/2301.12597 Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models . Preprint, arXiv:2301.12597

    2023 arXiv

  7. [15]

    Xudong Lin, Simran Tiwari, Shiyuan Huang, Manling Li, Mike Zheng Shou, Heng Ji, and Shih-Fu Chang. 2023. Towards fast adaptation of pretrained contrastive models for multi-channel video-language retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2023

  8. [16]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023 a . Improved baselines with visual instruction tuning

    2023

  9. [17]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 b . https://arxiv.org/abs/2304.08485 Visual instruction tuning . Preprint, arXiv:2304.08485

    2023 arXiv

  10. [18]

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. 2022 a . https://arxiv.org/abs/2206.08916 Unified-io: A unified model for vision, language, and multi-modal tasks . Preprint, arXiv:2206.08916

    2022 arXiv

  11. [19]

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. https://arxiv.org/abs/2310.02255 Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts . Preprint, a...

    2024 arXiv

  12. [20]

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022 b . Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing...

    2022

  13. [21]

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR)

    2019

  14. [22]

    Arvind Neelakantan, Quoc V Le, and Ilya Sutskever. 2015. Neural programmer: Inducing latent programs with gradient descent. arXiv preprint arXiv:1511.04834

    2015 arXiv

  15. [23]

    OpenAI. 2024. https://www.openai.com/gpt-4o Gpt-4o . Accessed: 2024-06-16

    2024

  16. [24]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 Bleu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311...

    2002

  17. [25]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. https://arxiv.org/abs/2103.00020 Learning transferable visual models from natural lan...

    2021 arXiv

  18. [26]

    Weimin Shi, Mingchen Zhuge, Dehong Gao, Zhong Zhou, Ming-Ming Cheng, and Deng-Ping Fan. 2023. https://arxiv.org/abs/2302.00952 Qr-clip: Introducing explicit open-world knowledge for location and time reasoning . Preprint, arXiv:2302.00952

    2023 arXiv

  19. [27]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lil...

    2024 arXiv

  20. [28]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.13971 Llama:...

    2023 arXiv

  21. [29]

    Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. 2019. Visual entailment: A novel task for fine-grained image understanding. arXiv preprint arXiv:1901.06706

    2019 arXiv

  22. [30]

    Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2019

  23. [31]

    Gengyuan Zhang, Yurui Zhang, Kerui Zhang, and Volker Tresp. 2024. Can vision-language models be a good guesser? exploring vlms for times and location reasoning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 636--645

    2024

  24. [32]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.