Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

VIST-GPT: Ushering in the Era of Visual Storytelling with LLMs?

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning a small multimodal LLM yields visual stories that sit closer to human narratives than prior models.

desk verdict A plausible fine-tuning recipe undermined by test-set tuning and an overclaimed benchmark—fix the protocol and it's a useful systems paper. read the letter →

arxiv 2504.19267 v3 pith:J7YAFU5V submitted 2025-04-27 cs.CL cs.AIcs.CVcs.LG

Mohamed Gado , Towhid Taliee , Muhammad Memon , Dmitry Ignatov , Radu Timofte This is my paper · ORCID
classification cs.CLcs.AIcs.CVcs.LG
keywords visualstorytellingVIST-GPTmultimodallargelanguagemodeldualencodergroundingreference-freeevaluationLoRAfine-tuningcoherence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that visual storytelling does not need a purpose-built architecture: a video-understanding multimodal LLM, fine-tuned on the VIST dataset, can turn a five-image sequence into a coherent five-sentence story. On the reference-free metrics GROOViST, RoViST, and UniEval, the resulting VIST-GPT v2 model scores highest among compared models for visual grounding, coherence, and fluency, and its stories sit closest to human-written ones by the Human-to-Machine Distance. The authors' central claim is that fine-tuning the language model while keeping the visual encoders frozen is enough to achieve strong visual grounding and to reduce hallucinated or irrelevant story details. This matters because n-gram metrics such as BLEU and CIDEr are poorly suited to storytelling, where many valid stories exist for the same image sequence.

What carries the argument

The load-bearing mechanism is the dual visual encoder pipeline inherited and adapted from VideoGPT+: CLIP ViT-L/14 supplies spatial features, InternVideo2 (an 8-frame video model) supplies temporal dynamics, trainable vision-language adapters project both into the LLM embedding space, and token pooling compresses the visual tokens. The Phi-3-mini-4k-instruct language model is then fine-tuned on VIST with LoRA while the visual side stays frozen, and at inference a task prompt plus low temperature and beam search enforce one sentence per image. The claim-carrying component is the fine-tuned LLM: it is what converts generic video understanding into story structure, and the comparison of v1 versus v2 shows that a more constraining prompt and decoding setup improves grounding and coherence.

What would settle it

A concrete falsifier: feed the model image sequences after shuffling or replacing the images while keeping the same prompt, and compare grounding; if GROOViST stays near 0.9962 when the images are replaced by unrelated images, the score does not measure visual grounding.

Watch

Extended reading notes

Core claim

VIST-GPT v2, built from the VideoGPT+ multimodal backbone with a CLIP ViT-L/14 image encoder, an InternVideo2 video encoder, and a Phi-3-mini-4k-instruct LLM fine-tuned with LoRA, establishes what the authors describe as a new benchmark for visual storytelling. On a 900-example intersection of VIST test predictions, it attains a GROOViST visual grounding score of 0.9962, a RoViST-C coherence score of 0.7837, and a UniEval fluency score of 0.950, with the lowest Human-to-Machine Distance (dHM = 0.0459) among AREL, GLACNET, KG Story, MCSM+BART, and VIST-GPT v1. The claim is that the model's narratives align with the objects and events in the image sequence, flow logically sentence to sentence, and avoid the hallucinated details that plague prior models, because the LLM was fine-tuned on story-level data rather than merely prompted.

Load-bearing premise

The load-bearing premise is that the frozen CLIP and InternVideo2 features, carried through the inherited adapters, contain enough visual information for the fine-tuned LLM to ground its stories; if they do not, the strong grounding scores could reflect language priors rather than image content.

Editorial extensions

If this is right

  • If the central claim holds, fine-tuning an existing video-centric multimodal LLM on VIST is a viable route to visual storytelling, without training a purpose-built encoder-decoder from scratch.
  • The high scores imply that frozen visual features are sufficient for story grounding, so future gains should come mostly from the language side or from decoding strategy.
  • The reference-free metrics RoViST and GROOViST, plus the Human-to-Machine Distance, give a reusable evaluation protocol that does not penalize valid but different stories, unlike BLEU or CIDEr.
  • VIST-GPT v2's lower dHM means that on this benchmark the model's stories share the grounding, coherence, and non-redundancy profile of human stories more closely than the four earlier storytelling models tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 900-example intersection may not represent the full VIST test set, and if baseline predictions are missing non-randomly, the reported margins could shift on a complete evaluation.
  • A text-only control experiment, feeding the same prompts with no visual tokens, would directly test whether the grounding scores reflect genuine image understanding or language priors learned from VIST.
  • Because the visual encoders are frozen, scaling the LLM or unfreezing the adapters on more diverse story data could push grounding further; that is a natural extension the authors do not run.
  • The qualitative appendix suggests the model captures social dynamics and emotional tone, but no metric in the paper measures those dimensions, so a reader should treat that as anecdotal.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents VIST-GPT, a visual storytelling model that adapts VideoGPT+'s dual image-video encoder architecture with a Phi-3-mini LLM fine-tuned via LoRA on the VIST dataset. The authors evaluate two inference variants (v1 and v2) on a 900-example intersection of the VIST test set using reference-free metrics (GROOViST, RoViST, UniEval) and the Human-to-Machine Distance (dHM). They claim that VIST-GPT v2 achieves state-of-the-art visual grounding, coherence, fluency, and the lowest dHM among compared models, thereby 'ushering in the era of visual storytelling with LLMs.' The technical contribution is an efficient fine-tuning recipe rather than a fundamentally new architecture.

Significance. If the empirical claims were sound, the paper would provide a useful, lightweight visual storytelling model with a sensible choice of reference-free evaluation metrics. The use of GROOViST and RoViST is appropriate for the task, and the efficiency of LoRA fine-tuning of a small LLM is attractive. However, the central benchmark claim is currently undermined by the evaluation protocol: the model version and decoding hyperparameters appear to have been selected on the test set itself, and the comparison omits several recent LLM-based storytelling baselines, including StoryLLaVA, which the paper cites in its own related work. Thus the significance of the contribution as a 'new benchmark' is not established, though the underlying methodology could be valid after a corrected evaluation.

major comments (3)
  1. [§4.2 and §5.3 (Tables 2–3)] The reported benchmark scores are invalid as test-set estimates because the model version was selected on the test set. Section 4.2 states that 'after completing the training process, we conducted several experiments during the inference phase, testing various parameters and prompts to optimize the model's performance,' resulting in v1 and v2. No validation split is described anywhere in Sections 4–5. The subsequent Tables 2 and 3 report GROOViST, RoViST-C, RoViST-NR, and dHM for v2 on the same 900-example test subset used to choose v2 over v1. Consequently, the values 0.9962, 0.7837, and 0.0459 are selection artifacts rather than estimates of generalization. The authors must either (a) introduce a proper validation split, tune only on it, and then report test results exclusively for the final version, or (b) clearly reframe these numbers as in-sample tuning results. This is load-bearing because the 'new benchmark' claim rests directly on these tables.
  2. [§3.3 vs §5.3] The comparison omits the most relevant modern baselines. Section 3.3 discusses StoryLLaVA as a recent visual storytelling model, yet Tables 2 and 3 compare only AREL, GLACNET, KG-Story, and MCSM+BART. The absence of StoryLLaVA and other LLM-based storytelling systems means that even with a clean evaluation, the paper would not substantiate 'v2 sets a new benchmark for visual storytelling.' The authors should include predictions from StoryLLaVA and, if feasible, from VideoGPT+ (the base model) and other recent MLLMs, with comparable inference settings. Without these, the claim is unsupported.
  3. [§4.1–4.2 and §5.2] The visual grounding results may be inflated by language priors because the visual encoders and adapters are frozen and only the LLM is fine-tuned. GROOViST measures alignment between nouns in the story and image regions; if the fine-tuned LLM produces generic nouns frequent in VIST training stories (e.g., 'friends,' 'park,' 'family'), the score can be high even when the model does not actually perceive the image content. The paper does not provide any diagnostic evidence to separate genuine visual grounding from language-prior effects. I recommend adding a per-image grounding analysis, or an experiment where the visual input is replaced with noise or a mismatched image sequence, to verify that the high GROOViST score is not an artifact of the text distribution. This caveat is important because the central claim includes 'highest visual grounding.'
minor comments (4)
  1. [§5.3, Eq. (1)–(2)] The dHM computation requires a human story H, but VIST has multiple human stories per image sequence. Please specify which human story is used (e.g., the first reference, a random reference, or an average over references) in the calculation of dH, dG, and dR; this affects the comparability of the distances across models.
  2. [§5.4] The UniEval scores reported in the text (mean coherence 0.855, understandability 0.9, fluency 0.950 for V2) are not shown in a table and are only given in Figure 5, which is not reproduced in the text. Please include a table with all model scores for all UniEval dimensions so that readers can reproduce the comparison.
  3. [§4.2] Please clarify how the additional three frames are padded when passing a five-image story to InternVideo2, which processes eight frames. Padding with zero frames versus repeating frames could affect temporal modeling and should be described precisely.
  4. [Throughout] The paper has numerous typos and formatting issues, including 'LL Ms' in the title, 'W e' in the abstract, 'judegment' in Section 5.2, and some incomplete figure captions. A thorough proofreading pass is needed.

Circularity Check

1 steps flagged · score 6.0 of 10

Benchmark claim is partially circular: VIST-GPT v2 was selected by inference-time prompt/decoding optimization on the same 900-example test set later reported as the benchmark.

  1. fitted input called prediction [Section 4.2, Table 1; Section 5.3, Tables 2 and 3]
    "After completing the training process, we conducted several experiments during the inference phase, testing various parameters and prompts to optimize the model's performance. This experimentation resulted in two distinct versions of the model, VIST-GPT v1 and VIST-GPT v2. ... To ensure a fair comparison, we obtained predictions for the VIST test set from previous models. Since the test set predictions were incomplete for all models, we took the intersection of the available prediction samples, resulting in 900 examples."

    The v2 configuration was selected by inference-time experimentation 'to optimize the model's performance' (Section 4.2), and the paper describes no validation or held-out split before reporting Tables 2 and 3. The only evaluation set introduced is the 900-example intersection of VIST test predictions. Reporting v2's scores on that same set as a 'new benchmark' therefore presents a selection artifact as an independent prediction: the chosen prompt, temperature, and beam count are inputs tuned on or toward the same examples whose scores are then reported as the model's performance. The reported superiority of v2 over v1 and over the baselines is not derived from a fixed, pre-specified model but is forced by the selection procedure, making the headline benchmark claim partially circular.

full rationale

Most of the technical derivation is not circular. The model adapts VideoGPT+'s frozen dual encoders and vision-language adapters, fine-tunes Phi-3-mini-4k-instruct with LoRA on VIST, and evaluates with externally defined metrics (RoViST, GROOViST, UniEval, dHM). These metrics are defined in prior work and are not functions of VIST-GPT's parameters, and no load-bearing claim rests solely on the authors' self-citations. The circular/fitted element is the benchmark claim itself: Section 4.2 describes tuning prompts and decoding parameters during inference to optimize performance, yielding v1 and v2, while Section 5 introduces no validation split and reports scores for the selected v2 on the same 900-example VIST test intersection. The headline 'new benchmark' therefore reduces, in part, to a selection statistic rather than an independent estimate of generalization. This fits the fitted-input-called-prediction pattern, though it is partial circularity because the architecture, training procedure, and metric definitions retain independent content.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the VIST data, the validity of the reference-free metrics, the adequacy of frozen visual features, and the representativeness of the 900-example comparison. None of these is established within the paper beyond citation; they are assumed.

free parameters (4)
  • VIST-GPT v1 temperature = 0.8
    Inference temperature chosen during experimentation; the variant was selected after inspecting test-set scores, so this value is tuned to the evaluation data.
  • VIST-GPT v1 beam count = 4
    Beam search width chosen during experimentation; variant selected on test-set performance.
  • VIST-GPT v2 temperature = 0.7
    Inference temperature for the better-performing variant; selection based on test-set metrics.
  • VIST-GPT v2 beam count = 2
    Beam search width for v2; selected after test-set evaluation.
assumptions (5)
  • domain assumption The VIST dataset provides image sequences with human-authored stories that serve as ground truth for storytelling quality.
    The paper trains and evaluates on VIST without questioning the validity of the human stories as the target (Sections 2, 5). If the human stories are noisy or unrepresentative, the evaluation is compromised.
  • domain assumption RoViST and GROOViST metrics are valid proxies for human judgment of visual grounding, coherence, and non-redundancy.
    The paper uses these metrics as the primary evidence of quality (Section 5.2), citing prior work [27,32] for their validity. The central claim depends on these metrics tracking human preferences.
  • domain assumption UniEval provides a valid measure of coherence, fluency, and understandability for storytelling.
    Section 5.4 uses UniEval without validating it on storytelling data; UniEval was designed for summarization and other NLG tasks [44].
  • domain assumption The frozen visual encoders and adapters inherited from VideoGPT+ capture sufficient visual information for story grounding.
    The visual side is not fine-tuned (Section 4.2); the model's grounding performance depends on the frozen CLIP and InternVideo2 features being adequate for the storytelling task.
  • domain assumption The 900-example intersection of available test predictions is representative of VIST test set performance.
    Section 5.3 compares models only on the intersection of prediction files from prior work; if the missing examples are biased, the comparison is unfair.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VIST-GPT: Ushering in the Era of Visual Storytelling with LLMs?." pith.science (2026). https://pith.science/paper/J7YAFU5V

@misc{pith2026250419267,
  author       = {Pith},
  title        = {Pith review of: VIST-GPT: Ushering in the Era of Visual Storytelling with LLMs?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J7YAFU5V}},
  note         = {Machine review of arXiv:2504.19267}
}
read the original abstract

Visual storytelling is an interdisciplinary field combining computer vision and natural language processing to generate cohesive narratives from sequences of images. This paper presents a novel approach that leverages recent advancements in multimodal models, specifically adapting transformer-based architectures and large multimodal models, for the visual storytelling task. Leveraging the large-scale Visual Storytelling (VIST) dataset, our VIST-GPT model produces visually grounded, contextually appropriate narratives. We address the limitations of traditional evaluation metrics, such as BLEU, METEOR, ROUGE, and CIDEr, which are not suitable for this task. Instead, we utilize RoViST and GROOVIST, novel reference-free metrics designed to assess visual storytelling, focusing on visual grounding, coherence, and non-redundancy. These metrics provide a more nuanced evaluation of narrative quality, aligning closely with human judgment.

Figures

Figures reproduced from arXiv: 2504.19267 by the authors.

Figure 1
Figure 1. A sample from the VIST dataset consists of five image [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. VIST-GPT Architecture (components in blue are fro [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of human-written story from the VIST da [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Human to Machine Distance dHM across different mod￾els (the less the better) 5.4. UniEval Evaluation Model dHM AREL 0.2403 GLACNET 0.1896 KG Story 0.1457 MCSM+BART 0.0976 VIST-GPT v1 0.0546 VIST-GPT v2 0.0459 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Comparison of coherence, understandability, and [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 5
Figure 5. Figure 5: Comparison of human-written and model-generated [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Comparison of human-written and model-generated [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Comparison of human-written and model-generated [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Comparison of human-written and model-generated [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Comparison of human-written and model-generated [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LEMUR 2: Unlocking Neural Network Diversity for AI

    cs.LG 2026-07 conditional novelty 5.5 of 10

    LEMUR 2 releases a multi-generator, multi-task neural-architecture corpus with real-device latency metadata intended as fuel for LLM-driven AutoML.

  2. AugmentGest: Can Random Data Cropping Augmentation Boost Gesture Recognition Performance?

    cs.CV 2025-06 conditional novelty 3.0 of 10

    Random cropping, rotation, zoom, and brightness/contrast augmentation on 2D skeleton gesture images improves accuracy on SHREC'17, DHG14/28, and JHMDB by up to 4.5% across three models.

Reference graph

Works this paper leans on

50 extracted references · 28 canonical work pages · cited by 2 Pith papers

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 3

    arXiv 2024

  2. [2]

    Augmentgest: Can random data cropping augmentation boost gesture recognition performance? arXiv preprint arXiv:2506.07216, 2025

    Nada Aboudeshish, Dmitry Ignatov, and Radu Timofte. Augmentgest: Can random data cropping augmentation boost gesture recognition performance? arXiv preprint arXiv:2506.07216, 2025. 8

    arXiv 2025

  3. [3]

    Commonsense knowledge aware concept selec- tion for diverse and informative visual storytelling

    Hong Chen, Yifei Huang, Hiroya Takamura, and Hideki Nakayama. Commonsense knowledge aware concept selec- tion for diverse and informative visual storytelling. In Pro- ceedings of the AAAI Conference on Artificial Intelligence , pages 999–1008, 2021. 1, 2, 6

    work page 2021

  4. [4]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Y onghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See 8 https://vicuna. lmsys. org (accessed 14 April 2023) , 2(3):6,

    work page 2023

  5. [5]

    Content Planning for Neural Story Generation with Aristotelian Rescoring

    Seraphina Goldfarb-Tarrant, Tuhin Chakrabarty, Ralph Weischedel, and Nanyun Peng. Content planning for neural story generation with aristotelian rescoring. arXiv preprint arXiv:2009.09870, 2020. 2

    work page Pith review arXiv 2009

  6. [6]

    Contex - tualize, show and tell: A neural visual storyteller

    Diana Gonzalez-Rico and Gibran Fuentes-Pineda. Contex - tualize, show and tell: A neural visual storyteller. arXiv preprint arXiv:1806.00738, 2018. 1, 2

    arXiv 2018

  7. [7]

    Lemur neural net- work dataset: Towards seamless automl

    Arash Torabi Goodarzi, Roman Kochnev, Waleed Khalid, Furui Qin, Tolgay Atinc Uzun, Yashkumar Sanjaybhai Dhameliya, Yash Kanubhai Kathiriya, Zofia Antonina Ben- tyn, Dmitry Ignatov, and Radu Timofte. Lemur neural net- work dataset: Towards seamless automl. arXiv preprint arXiv:2504.10552, 2025. 2

    arXiv 2025

  8. [8]

    Diverse and rel- evant visual storytelling with scene graph embeddings

    Xudong Hong, Rakshith Shetty, Asad Sayeed, Khushboo Mehra, V era Demberg, and Bernt Schiele. Diverse and rel- evant visual storytelling with scene graph embeddings. In Proceedings of the 24th Conference on Computational Nat- ural Language Learning, pages 420–430, 2020. 1, 2

    work page 2020

Show all 50 references
  1. [9]

    Knowledge-enriched visual storytelling

    Chao-Chun Hsu, Zi-Y uan Chen, Chi-Yang Hsu, Chih-Chia Li, Tzu-Y uan Lin, Ting-Hao Huang, and Lun-Wei Ku. Knowledge-enriched visual storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 7952– 7960, 2020. 2, 6

    2020

  2. [10]

    Plot and rework: Modeling storylines for visual storytelling

    Chi-Yang Hsu, Y un-Wei Chu, Ting-Hao’Kenneth’ Huang, and Lun-Wei Ku. Plot and rework: Modeling storylines for visual storytelling. arXiv preprint arXiv:2105.06950 , 2021. 2

    2021 arXiv

  3. [11]

    Visual storytelling

    Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh , Is- han Misra, Aishwarya Agrawal, Jacob Devlin, Ross Gir- shick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. Visual storytelling. In Proceedings of the 2016 conference of the North American chapter of the association ...

    2016

  4. [12]

    Glac net: Glocal attention cascading networks for multi-image cued story generation

    Taehyeong Kim, Min-Oh Heo, Seonil Son, Kyoung-Wha Park, and Byoung-Tak Zhang. Glac net: Glocal attention cascading networks for multi-image cued story generation. arXiv preprint arXiv:1805.10973, 2018. 1, 2, 6

    2018 arXiv

  5. [13]

    Optuna vs code llama: Are llms a new paradigm for hyperparameter tuning? arXiv preprint arXiv:2504.06006, 2025

    Roman Kochnev, Arash Torabi Goodarzi, Zofia Antonina Bentyn, Dmitry Ignatov, and Radu Timofte. Optuna vs code llama: Are llms a new paradigm for hyperparameter tuning? arXiv preprint arXiv:2504.06006, 2025. 2

    2025 arXiv

  6. [14]

    Nngpt: Neural network model gener- ation

    Roman Kochnev et al. Nngpt: Neural network model gener- ation. arXiv preprint, 2025. 2

    2025

  7. [15]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 2

    2023

  8. [16]

    A character- centric neural model for automated story generation

    Danyang Liu, Juntao Li, Meng-Hsuan Y u, Ziming Huang, Gongshen Liu, Dongyan Zhao, and Rui Yan. A character- centric neural model for automated story generation. In Pro- ceedings of the AAAI Conference on Artificial Intelligence , pages 1725–1732, 2020. 1, 2

    2020

  9. [17]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Y ong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2, 3

    2024

  10. [18]

    Visual relationship detection with language priors

    Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei- Fei. Visual relationship detection with language priors. I n Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Pro- ceedings, Part I 14, pages 852–869. Springer, 2016. 2

    2016

  11. [19]

    Videogpt+: Integrating image and video en- coders for enhanced video understanding

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Videogpt+: Integrating image and video en- coders for enhanced video understanding. arXiv preprint arXiv:2406.09418, 2024. 3

    2024 arXiv

  12. [20]

    Step-by- step: Separating planning from realization in neural data- to- text generation

    Amit Moryossef, Y oav Goldberg, and Ido Dagan. Step-by- step: Separating planning from realization in neural data- to- text generation. arXiv preprint arXiv:1904.03396, 2019. 2

    1904 arXiv

  13. [21]

    Planning with learned entity prompts for abstractive summarization

    Shashi Narayan, Yao Zhao, Joshua Maynez, Gonc ¸alo Sim˜ oes, Vitaly Nikolaev, and Ryan McDonald. Planning with learned entity prompts for abstractive summarization . Transactions of the Association for Computational Linguis - tics, 9:1475–1492, 2021. 2

    2021

  14. [22]

    Data-to - text generation with variational sequential planning

    Ratish Puduppully, Yao Fu, and Mirella Lapata. Data-to - text generation with variational sequential planning. Trans- actions of the Association for Computational Linguistics, 10: 697–715, 2022. 2

    2022

  15. [23]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  16. [24]

    Plotmachines: Outline-conditioned generation with d y- namic plot state tracking

    Hannah Rashkin, Asli Celikyilmaz, Yejin Choi, and Jian feng Gao. Plotmachines: Outline-conditioned generation with d y- namic plot state tracking. arXiv preprint arXiv:2004.14967,

    2004 arXiv

  17. [25]

    Explo r- ing the collaboration between vision models and llms for en- hanced image classification

    Bhavya Rupani, Dmitry Ignatov, and Radu Timofte. Explo r- ing the collaboration between vision models and llms for en- hanced image classification. Dimensions, 27(1), 2025. 2

    2025

  18. [26]

    Storygpt-v: Larg e language models as consistent story visualizers

    Xiaoqian Shen and Mohamed Elhoseiny. Storygpt-v: Larg e language models as consistent story visualizers. 2023. 1

    2023

  19. [27]

    Groovist: A metric for grounding objects in visual story- telling

    Aditya K Surikuchi, Sandro Pezzelle, and Raquel Fern´ andez. Groovist: A metric for grounding objects in visual story- telling. arXiv preprint arXiv:2310.17770, 2023. 6

    2023 arXiv

  20. [28]

    Not (yet) the whole story: Evaluating visual storytelling r e- quires more than measuring coherence, grounding, and rep- etition

    Aditya K Surikuchi, Raquel Fern´ andez, and Sandro Pezzelle. Not (yet) the whole story: Evaluating visual storytelling r e- quires more than measuring coherence, grounding, and rep- etition. arXiv preprint arXiv:2407.04559, 2024. 6

    2024 arXiv

  21. [29]

    V okenization: Improving lan- guage understanding with contextualized, visual-grounde d supervision

    Hao Tan and Mohit Bansal. V okenization: Improving lan- guage understanding with contextualized, visual-grounde d supervision. arXiv preprint arXiv:2010.06775, 2020. 2

    2010 arXiv

  22. [30]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 2

    2023 arXiv

  23. [31]

    Multimodal few-shot learning with frozen language models

    Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Es- lami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200–212, 2021. 2 9

    2021

  24. [32]

    Rovist: Learn- ing robust metrics for visual storytelling

    Eileen Wang, Caren Han, and Josiah Poon. Rovist: Learn- ing robust metrics for visual storytelling. arXiv preprint arXiv:2205.03774, 2022. 5, 6

    2022 arXiv

  25. [33]

    Storytelling from an image stream using scene graphs

    Ruize Wang, Zhongyu Wei, Piji Li, Qi Zhang, and Xuan- jing Huang. Storytelling from an image stream using scene graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9185–9192, 2020. 2

    2020

  26. [34]

    No metrics are perfect: Adver- sarial reward learning for visual storytelling

    Xin Wang, Wenhu Chen, Y uan-Fang Wang, and William Yang Wang. No metrics are perfect: Adver- sarial reward learning for visual storytelling. arXiv preprint arXiv:1804.09160, 2018. 6

    2018 arXiv

  27. [35]

    Internvideo2: Scaling video foundation mod- els for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Y u, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation mod- els for multimodal video understanding. arXiv preprint arXiv:2403.15377, 2024. 3

    2024 arXiv

  28. [36]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519, 2023. 2

    2023 arXiv

  29. [37]

    Imagine, reason and write: Visual storytelling with graph knowledge and relational reasonin g

    Chunpu Xu, Min Yang, Chengming Li, Ying Shen, Xiang Ao, and Ruifeng Xu. Imagine, reason and write: Visual storytelling with graph knowledge and relational reasonin g. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 3022–3029, 2021. 1, 2

    2021

  30. [38]

    A skeleton-based model for promoting co- herence among sentences in narrative story generation

    Jingjing Xu, Xuancheng Ren, Yi Zhang, Qi Zeng, Xiaoyan Cai, and Xu Sun. A skeleton-based model for promoting co- herence among sentences in narrative story generation. arXiv preprint arXiv:1808.06945, 2018. 2

    2018 arXiv

  31. [39]

    Re3: Generating longer stories with recursive reprompting and revision

    Kevin Yang, Y uandong Tian, Nanyun Peng, and Dan Klein. Re3: Generating longer stories with recursive reprompting and revision. arXiv preprint arXiv:2210.06774, 2022. 1, 2

    2022 arXiv

  32. [40]

    StoryLLaV A: Enhancing visual storytelling with multi- modal large language models

    Li Yang, Zhiding Xiao, Wenxin Huang, and Xian Zhong. StoryLLaV A: Enhancing visual storytelling with multi- modal large language models. In Proceedings of the 31st In- ternational Conference on Computational Linguistics , pages 3936–3951, Abu Dhabi, UAE, 2025. Association for ...

    2025

  33. [41]

    Knowledgeable storyteller: A commonsense-driven generative model for visual story- telling

    Pengcheng Yang, Fuli Luo, Peng Chen, Lei Li, Zhiyi Yin, Xiaodong He, and Xu Sun. Knowledgeable storyteller: A commonsense-driven generative model for visual story- telling. In IJCAI, page 7, 2019. 2

    2019

  34. [42]

    Plan-and-write: Towards bet- ter automatic storytelling

    Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. Plan-and-write: Towards bet- ter automatic storytelling. In Proceedings of the AAAI Con- ference on Artificial Intelligence , pages 7378–7385, 2019. 2

    2019

  35. [43]

    Minigpt- 5: Interleaved vision-and-language generation via genera tive vokens

    Kaizhi Zheng, Xuehai He, and Xin Eric Wang. Minigpt- 5: Interleaved vision-and-language generation via genera tive vokens. arXiv preprint arXiv:2310.02239, 2023. 2

    2023

  36. [44]

    To- wards a unified multi-dimensional evaluator for text genera - tion

    Ming Zhong, Yang Liu, Da Yin, Y uning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. To- wards a unified multi-dimensional evaluator for text genera - tion. arXiv preprint arXiv:2210.07197, 2022. 7

    2022 arXiv

  37. [45]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 2 10 VIST-GPT: Ushering in the Era of Visual Storytelling with LL Ms? Supplemen...

    2023 arXiv

  38. [46]

    Appendix 8.1. Qualitative Evaluation To assess the effectiveness of VIST-GPT v2 in visual sto- rytelling, we compared its outputs with those of baseline models (AREL, GLACNET, kG Story, MCSM, and VIST- GPT v1), as well as human-generated stories, using two im- age sets: a skat...

  39. [47]

    and a family boating trip (Figure 6). 8.1.1. Skatepark Story The skateboarding image set (Figure

  40. [48]

    ollie.” The phrase “nailed it!

    shows two friends performing tricks at a park. The human story effectively captures the scene’s energy and camaraderie, detailing spe - cific skateboarding tricks and expressing the narrator’s ex - citement. In contrast, baseline models struggled to reflect the activity depicted...

  41. [49]

    The human story pro- vides a rich description of the family’s shared experience, including interactions and conversation, accurately capt ur- ing the sense of warmth and connection

    depict a relaxed multi-generational outing on a lake. The human story pro- vides a rich description of the family’s shared experience, including interactions and conversation, accurately capt ur- ing the sense of warmth and connection. Baseline models, however, frequently dive...

  42. [50]

    the daughter wouldn’t leave them alone,

    captures the atmo- sphere of a family gathering, focusing on drinking, playing video games, and transitioning to outdoor activities like a game of catch. It highlights group dynamics and the natu- ral flow of events, creating an engaging and contextually rich narrative. In cont...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.