Pith. sign in

REVIEW 2 major objections 5 minor 45 references

ViSTa Dataset: Do vision-language models understand sequential tasks?

T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that current vision-language models, including GPT-4o, cannot reliably understand multi-step sequential tasks, despite strong object recognition.

desk verdict ViSTa is a genuinely useful benchmark for testing sequential understanding in VLMs, but the headline claim is not fully supported because the virtual-home videos are stitched from different ALFRED episodes and no control verifies that stitching artifacts aren't driving the order-related failures. read the letter →

arxiv 2411.13211 v2 pith:Q37YGDT2 submitted 2024-11-20 cs.CV cs.LG

Evžen Wybitul , Evan Ryan Gunter , Mikhail Seleznyov , David Lindner This is my paper · ORCID
classification cs.CVcs.LG
keywords vision-languagemodelsrewardsequentialtaskunderstandingactionordervideo-languagebenchmarkreinforcementlearningprocesssupervision
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether vision-language models (VLMs) can serve as reward models for tasks that cannot be scored by the final state alone, meaning whether they genuinely understand the step-by-step order of a task. It introduces ViSTa, a dataset of more than 4,000 videos in virtual home, Minecraft, and real-world environments, organized into eight levels of increasing sequential complexity. Evaluations of CLIP, ViCLIP, and GPT-4o show that all three recognize objects well but struggle with sequences: on permutation problems, where only the order of actions differs, GPT-4o's performance drops sharply as videos lengthen, while the other models stay near the majority-class baseline. The paper concludes that current VLMs are not yet capable of supervising any but the most basic tasks.

What carries the argument

ViSTa's hierarchical structure is the central mechanism: level-1 single-action videos are composed into levels 2 through 8, and video-description pairs are grouped into problem sets that test object recognition, object properties, action understanding, and, through permutation problem sets, action order. The scoring protocol samples 8 to 32 frames per video, standardizes softmax scores across classes, and picks the highest-scoring description. The permutation problem sets are the load-bearing part of the benchmark because they keep objects and actions the same and vary only their order.

What would settle it

Have human annotators watch the automatically stitched virtual home permutation videos and try to identify the correct action order. If humans cannot reliably do so, the videos do not contain the order information ViSTa claims to measure, and the reported model failures would reflect dataset artifacts rather than a lack of sequential understanding.

Watch

Extended reading notes

Core claim

On ViSTa's permutation problem sets, the only tested model that beats the majority-class baseline is GPT-4o, and its macro F1 begins to fall after level 4, ending at roughly half of its original value for eight-action videos. CLIP and ViCLIP stay barely above baseline throughout. In general multi-step problem sets, CLIP can match videos to descriptions without any sense of order, which the paper attributes to the models relying on object recognition rather than on action order. The paper's central claim is that current VLMs cannot yet supervise sequential tasks and lack robust action-order understanding.

Load-bearing premise

The load-bearing assumption is that the automatically stitched virtual home videos in ViSTa, which contain visual glitches such as sudden teleportation and changes in held objects, still carry valid, human-recoverable action-order information; the paper does not verify this.

Editorial extensions

If this is right

  • VLM-based process reward models are not yet viable for reinforcement learning on multi-step tasks, because the model cannot reliably tell whether steps happened in the correct order.
  • Evaluation of task understanding must separate order understanding from object recognition; ViSTa's permutation problem sets do this, while general problem sets do not.
  • Improving frame rate or model scale may improve general task matching, but the permutation results show that these levers do not fix order understanding for the tested models.
  • For now, a frontier model like GPT-4o can be considered for supervision only of single-step or very short tasks, and its drop on eight-action videos shows why.
  • If the failures are general, benchmark designers should treat 'matches the objects' and 'understands the sequence' as distinct capabilities rather than one skill.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to fine-tune a VLM on order-contrastive pairs like ViSTa's permutation sets; success on held-out environments would suggest the failure is a training gap rather than a fundamental architectural limit.
  • The finding that general multi-step problems can be solved without order understanding implies that earlier VLM reward-model successes on trajectory-style tasks may have been driven by object recognition; this could be checked by testing those reward models on wrong-order videos with identical objects.
  • Because the glitch concern applies to the largest split, the paper's strongest evidence for the conclusion comes from the smaller real-world and Minecraft sets; scaling those up would directly test whether the virtual-home results generalize.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces ViSTa, a hierarchical dataset of over 4,000 videos with step-by-step task descriptions in virtual home, Minecraft, and real-world environments. The dataset is organized into problem sets and levels, enabling evaluation of vision-language models on single-action and multi-action sequential tasks. The authors evaluate CLIP, ViCLIP, and GPT-4o on video-description matching, finding that all models perform well on object recognition but poorly on action-order understanding, with GPT-4o showing non-trivial but declining performance on longer sequences. The central claim is that current VLMs are not yet capable of supervising sequential tasks, with implications for using VLMs as process reward models in reinforcement learning.

Significance. If the results hold, ViSTa would be a valuable benchmark for studying VLM capabilities on process-based supervision, complementing existing outcome-based reward-model evaluations. The hierarchical structure and the separation of object recognition, object properties, actions, general sequential understanding, and permutation-based order understanding are thoughtful design choices. The paper also tests models across three environments, including real-world videos that are human-validated, and provides open-source code and data. The strongest evidence is the consistent near-baseline performance of CLIP and ViCLIP on permutation problems and the decline of GPT-4o with level, which would support the conclusion that current VLMs cannot reliably supervise multi-step tasks.

major comments (2)
  1. [3.2 and A.1.2] The virtual-home multiple-action videos are created by stitching clips from different ALFRED episodes, and the paper asserts that the resulting 'small visual glitches' are 'not problematic for our evaluations, since the models all have low frame-rates.' This assertion is load-bearing because the majority of ViSTa consists of these stitched videos. A low frame rate does not remove semantic mismatches (e.g., a pick-up clip and a put-down clip of a banana from different episodes can depict different objects), nor does it prevent stitch boundaries from making segment order artificially salient. No control is provided: there is no comparison between stitched videos and continuous re-renderings of the same tasks, no human baseline on the virtual-home data, and no analysis of whether model errors concentrate on videos with detectable discontinuities. Without such a control, the level-dependent drop in Fig. 3b and Fig. 13a may reflect video-construction artifacts rather than model limitations in understanding action order.
  2. [A.1.4 and Author contribution statement] Human validation at 100% accuracy is claimed only for the real-world videos, not for the virtual-home or Minecraft data. Since the virtual-home videos are automatically generated and constitute the majority of the dataset, the ground truth for those videos is not independently verified. If the stitched videos are ambiguous even to humans, the measured 'failure' of VLMs would be an artifact of the dataset rather than a model deficiency. The paper should provide a human baseline on a sample of virtual-home videos (and ideally Minecraft) to establish that the intended task is actually entailed by the videos.
minor comments (5)
  1. [Abstract and Section 5] The phrase 'they fail to understand sequential tasks' is stronger than the reported results, which show GPT-4o achieving non-trivial (though declining) performance on permutation problems, while CLIP and ViCLIP are near baseline. Consider softening the wording to 'models are not yet reliable at supervising sequential tasks' to match the evidence.
  2. [Section 4.1, Eq. (1)] The score standardization computes µ and σ from the evaluation videos themselves, which could in principle leak information about the test set. The footnote says results did not meaningfully differ when excluding the video being tested, but this leave-one-out variant should be reported in the main text or appendix to make the protocol unambiguous.
  3. [Section 4.2 and Figure 17] The claim that 'real videos are easier than simulations' rests on a comparison of only 9 real-world videos against an unspecified number of virtual-home videos. This is a small sample, and the conclusion should be hedged accordingly.
  4. [Section 4.2 and Figure 5] The attribution of CLIP-8's advantage over ViCLIP to 'larger scale' is speculative, since the models differ in architecture, training data, and other factors beyond parameter count. Please phrase this as a hypothesis rather than a conclusion.
  5. [Appendix C, Figure 10] The prompt text in Figure 10 contains a typo ('sligthly' should be 'slightly').

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ViSTa is an external evaluation instrument, and the reported VLM failures are measured against it rather than derived from its construction.

full rationale

The paper's central claim is an empirical evaluation result: ViSTa videos are scored by CLIP, ViCLIP, and GPT-4o against problem-set descriptions, and performance is reported as macro F1. No parameter or label is fitted to the target conclusion. The only preprocessing step, standardization in Sec. 4.1, normalizes per-class model scores with a mean and standard deviation, and the footnote explicitly checks leave-one-out robustness; it does not encode the level hierarchy or the action-order ground truth into the scores. The hierarchical design (levels 1-8) is an experimental manipulation, and the finding that longer tasks are harder is read off the models' outputs, so it is not true by construction. The one overlapping-author reference, Rocamonde et al. [8], is used only as background motivation for VLM-based reward models and is never invoked to justify the ViSTa evaluation or the failure conclusions. The stitching-discontinuity discussion in Sec. A.1.2 is a data-validity assumption about the stimulus ('small visual glitches... are not problematic for our evaluations, since the models all have low frame-rates'), but that is an unverified empirical claim about the videos, not a circular derivation: even if the glitches confound the measurement, the evaluation would still be an external measurement of model behavior. No step in the paper's derivation chain reduces to its own inputs, so there is no circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The evaluation rests on several domain assumptions about what forced-choice matching measures, the validity of automatically stitched videos, and the correctness of labels in the virtual home and Minecraft subsets. No formal axioms are introduced; the assumptions are empirical and testable.

free parameters (4)
  • Score standardization statistics mu and sigma = computed from evaluation videos, per class
    Section 4.1 standardizes final scores using mean and standard deviation computed from the same videos; the authors state this greatly improves performance. This is a post-hoc normalization choice that can alter measured accuracy.
  • Number of input frames = GPT-4o: 16; CLIP: 32; ViCLIP: 8
    Section 4.1 sets frame counts per model. Figure 16 shows CLIP performance rises with frame rate in some task groups, so this choice affects results; ViCLIP is fixed at 8 by the model.
  • GPT-4o prompting scheme = two-conversation, one-shot, exactly-one-highest-score
    Appendix B describes the hand-designed two-stage pipeline that first generates frame descriptions and then matches them to candidate labels. The authors state it outperformed a single-pass scheme, but no text-only control separates visual from language failures.
  • Minecraft action taxonomy = 7 basic actions: place, break, craft, combat, find, mine, pick up
    Section A.1.3 defines the action set chosen by the authors, inspired by Yuan et al. The composition of the 53-video Minecraft subset depends on this choice.
assumptions (3)
  • domain assumption Forced-choice video-description matching is a valid measure of sequential task understanding
    The entire evaluation operationalizes understanding as choosing among a small set of descriptions. The paper does not validate this against open-ended supervision or against the scoring needs of RL reward models.
  • domain assumption Visual discontinuities in remixed ALFRED videos do not affect VLM judgments
    Section A.1.2 asserts that low frame rates make stitching glitches undetectable, but no experiment verifies that the glitches do not destroy or leak order information.
  • domain assumption Ground-truth descriptions are correct and unambiguous for virtual home and Minecraft videos
    Human 100% accuracy is asserted only for real-world videos in Section A.1.4. No human baseline is reported for virtual home or Minecraft, so label solvability is not established for the largest portion of the dataset.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ViSTa Dataset: Do vision-language models understand sequential tasks?." pith.science (2026). https://pith.science/paper/Q37YGDT2

@misc{pith2026241113211,
  author       = {Pith},
  title        = {Pith review of: ViSTa Dataset: Do vision-language models understand sequential tasks?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q37YGDT2}},
  note         = {Machine review of arXiv:2411.13211}
}
read the original abstract

Using vision-language models (VLMs) as reward models in reinforcement learning holds promise for reducing costs and improving safety. So far, VLM reward models have only been used for goal-oriented tasks, where the agent must reach a particular final outcome. We explore VLMs' potential to supervise tasks that cannot be scored by the final state alone. To this end, we introduce ViSTa, a dataset for evaluating Vision-based understanding of Sequential Tasks. ViSTa comprises over 4,000 videos with step-by-step descriptions in virtual home, Minecraft, and real-world environments. Its novel hierarchical structure -- basic single-step tasks composed into more and more complex sequential tasks -- allows a fine-grained understanding of how well VLMs can judge tasks with varying complexity. To illustrate this, we use ViSTa to evaluate state-of-the-art VLMs, including CLIP, ViCLIP, and GPT-4o. We find that, while they are all good at object recognition, they fail to understand sequential tasks, with only GPT-4o achieving non-trivial performance.

Figures

Figures reproduced from arXiv: 2411.13211 by the authors.

Figure 1
Figure 1. ViSTa is a hierarchical dataset of videos with step-by-step descriptions. ViSTa enables granular testing of task sequences in three different environments. Tasks are organized by number of sub-tasks into a hierarchy of 8 levels. Videos within levels are grouped into problem sets (fig. 2) testing specific capabilities. ∗ ewybitul@ethz.ch † See author contribution statement. 1 https://github.com/Eugleo/vista-dataset a… view at source ↗
Figure 2
Figure 2. A problem set for action-order understanding. Problem sets are groups of videos to be matched with their descriptions. Each set targets a specific capability, e.g. understanding action order. 2 Related work VLMs as reward models. Previous studies have extensively explored using vision [6, 11, 12], language [13, 14], and vision-language models [8, 9, 10, 14, 15, 16, 17] for success detection in outcome-based tasks. F… view at source ↗
Figure 3
Figure 3. Macro F1 score averaged over groups of problem sets in level 1 (fig. 3a) and in higher levels (fig. 3b). Error ranges are 95% C.I. In multiple-step tasks, we see a gradual drop in performance with rising problem set level (fig. 3b). We find the following: • Understanding action order is very hard. This can most easily be seen when we look at the subset of multiple-action problem sets that test action order understan… view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Understanding action order in long videos is hard. In permutation problems, which focus on testing action-order understanding, GPT-4o’s performance starts dropping after level 4, ending up at around 50% of its original value for videos with 8 actions. This is not great…
Figure 5
Figure 5. Figure 5: Frame rate and model scale play an important role in general sequential tasks. We see that the performance of CLIP rises with increasing frame rate. When we compare CLIP-8 and ViCLIP, which both get 8 frames, we see that CLIP-8 nevertheless does much better. This is li…
Figure 6
Figure 6. Figure 6: The system prompt, used in both requests. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: The prompt used to obtain frame-by-frame descriptions. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: The prompt used to make GPT-4o think step by step about how well each class description [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: The prompt used to obtain the final scores in a predictable format. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: The prompt used to obtain frame-by-frame descriptions for Minecraft videos. [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: The prompt used to obtain the final scores in a predictable format for Minecraft videos in [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: The prompt used to obtain the final scores in a predictable format for Minecraft videos in [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Performance overview in the virtual home environment. Ranges are 95% C.I. [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Performance overview in the real world environment. Ranges are 95% C.I. [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Performance overview in the Minecraft environment. Ranges are 95% C.I. [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Frame rate does not play a large role in level 1 tasks. The performance of CLIP and ViCLIP on different groups of problem sets in the virtual home environment, after changing the number of CLIP input frames. Ranges are 95% C.I. 20 [PITH_FULL_IMAGE:figures/full_fig_p0…
Figure 17
Figure 17. Figure 17: Performance on identifying which object was picked up from a cluttered table in the virtual [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 22 canonical work pages

  1. [1]

    Mas- tering the game of go with deep neural networks and tree search

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016

    2016

  2. [2]

    Deep reinforcement learning for robotics: A survey of real-world successes

    Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Martín-Martín, and Peter Stone. Deep reinforcement learning for robotics: A survey of real-world successes. arXiv preprint arXiv:2408.03539, 2024

    arXiv 2024

  3. [3]

    Specification gaming: the flip side of ai ingenuity

    Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of ai ingenuity. DeepMind Blog, 3, 2020

    work page 2020

  4. [4]

    Defining and characterizing reward gaming

    Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460– 9471, 2022. 6

    work page 2022

  5. [5]

    On the importance of hyperparameter optimization for model-based reinforcement learning

    Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp, Kurtland Chua, Frank Hutter, and Roberto Calandra. On the importance of hyperparameter optimization for model-based reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 4015–4023. PMLR, 2021

    work page 2021

  6. [6]

    Variational inverse control with events: A general framework for data-driven reward definition

    Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. Advances in neural information processing systems, 31, 2018

    work page 2018

  7. [7]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  8. [8]

    Vision- Language Models are Zero-Shot Reward Models for Reinforcement Learning, 2023

    Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision- Language Models are Zero-Shot Reward Models for Reinforcement Learning, 2023. URL http://arxiv.org/abs/2310.12921

    arXiv 2023

Show all 45 references
  1. [9]

    Sontakke, Jesse Zhang, Sébastien M

    Sumedh A. Sontakke, Jesse Zhang, Sébastien M. R. Arnold, Karl Pertsch, Erdem Bıyık, Dorsa Sadigh, Chelsea Finn, and Laurent Itti. RoboCLIP: One Demonstration is Enough to Learn Robot Policies, 2023. URL http://arxiv.org/abs/2310.07899

    2023 arXiv

  2. [10]

    Task Success

    Lin Guan, Yifan Zhou, Denis Liu, Yantian Zha, Heni Ben Amor, and Subbarao Kambhampati. "Task Success" is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviors, 2024. URL http://arxiv.org/abs/ 2402.04210

    2024 arXiv

  3. [11]

    in-the-wild

    Annie S Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from" in-the-wild" human videos. arXiv preprint arXiv:2103.16817, 2021

    2021 arXiv

  4. [12]

    The unsur- prising effectiveness of pre-trained vision models for control

    Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsur- prising effectiveness of pre-trained vision models for control. In international conference on machine learning, pages 17359–17371. PMLR, 2022

    2022

  5. [13]

    Reward design with language models

    Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. arXiv preprint arXiv:2303.00001, 2023

    2023 arXiv

  6. [14]

    Smith, and Han- naneh Hajishirzi

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khy- athi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Han- naneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling. CoRR, abs/2403.13787, 2024....

    2024 doi arXiv

  7. [15]

    Reward learning from narrated demonstrations

    Hsiao-Yu Tung, Adam W Harley, Liang-Kang Huang, and Katerina Fragkiadaki. Reward learning from narrated demonstrations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7004–7013, 2018

    2018

  8. [16]

    Zero-shot reward specification via grounded natural language

    Parsa Mahmoudieh, Deepak Pathak, and Trevor Darrell. Zero-shot reward specification via grounded natural language. In International Conference on Machine Learning, pages 14743– 14752. PMLR, 2022

    2022

  9. [17]

    RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback, 2024

    Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback, 2024. URL http://arxiv.org/abs/2402.03681

    2024 arXiv

  10. [18]

    PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain, 2024

    Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Xiangdi Meng, Tianyu Liu, and Baobao Chang. PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain, 2024. URL http://arxiv.org/ abs/2402.15527

    2024 arXiv

  11. [19]

    Paxion: Patching Action Knowledge in Video-Language Foundation Models,

    Zhenhailong Wang, Ansel Blume, Sha Li, Genglin Liu, Jaemin Cho, Zineng Tang, Mohit Bansal, and Heng Ji. Paxion: Patching Action Knowledge in Video-Language Foundation Models,

  12. [20]

    Matthews, Ivan Villa-Renteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Yunzhu Li, Silvio Savarese, Hyowon Gweon, C

    Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Wensi Ai, Benjamin Martinez, Hang Yin, Michael Lingelbach, Minjune Hwang, Ayano Hiranaka, Sujay Garlanka, Arman Aydin, Sharon Lee, Jiankai Sun, Mona Anvar...

    2024 arXiv

  13. [21]

    VirtualHome: Simulating Household Activities via Programs, 2018

    Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. VirtualHome: Simulating Household Activities via Programs, 2018. URL http: //arxiv.org/abs/1806.07011

    2018 arXiv

  14. [22]

    Habitat: A Platform for Embodied AI Research, 2019

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research, 2019. URL http://arxiv.org/abs/1904. 01201

    2019

  15. [23]

    Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottaghi

    Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander William Clegg, Michal Hlavac, So Yeon Min, Vladimír V ondruš, Theophile Gervet, Vincent-Pierre Berges, John M. Turner, Oleksandr Maksymets, Zsolt Kira, Mrina...

    2023 arXiv

  16. [24]

    ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks, 2020

    Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks, 2020. URL http://arxiv.org/abs/1912.01734

    2020 arXiv

  17. [25]

    A short note on the kinetics- 700 human action dataset

    Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics- 700 human action dataset. arXiv preprint arXiv:1907.06987, 2019

    1907 arXiv

  18. [26]

    BEDD: the minerl BASALT evaluation and demonstrations dataset for training and benchmarking agents that solve fuzzy tasks

    Stephanie Milani, Anssi Kanervisto, Karolis Ramanauskas, Sander Schulhoff, Brandon Houghton, and Rohin Shah. BEDD: the minerl BASALT evaluation and demonstrations dataset for training and benchmarking agents that solve fuzzy tasks. In Alice Oh, Tris- tan Naumann, Amir Globerso...

    2023

  19. [27]

    Plan4mc: Skill reinforcement learning and planning for open-world minecraft tasks

    Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongqing Lu. Plan4mc: Skill reinforcement learning and planning for open-world minecraft tasks. CoRR, abs/2303.16563, 2023. doi: 10.48550/ARXIV .2303.16563. URL https://doi.org/10. 48550/arXiv.2303.16563

    2023 doi arXiv

  20. [28]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  21. [29]

    Reproducible scaling laws for contrastive language-image learning, 2022

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning, 2022. URL http://arxiv.org/abs/2212.07143

    2022 arXiv

  22. [30]

    LAION-5B: An open large-scale dataset for training next generation image-text models, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION...

    2022 arXiv

  23. [31]

    InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation, 2024

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, Conghui He, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, and Yu Qiao. InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation, 2024. UR...

    2024 arXiv

  24. [32]

    break-craft

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 9 Author contribution statement Evžen Wybitul: • Implemented the current ...

    2024 arXiv

  25. [34]

    First conversation, to get frame-by-frame descriptions: (a) First and only message: system prompt (figure 6) + frame-by-frame prompt (figure 7)

  26. [35]

    likely does not describe the video

    In a second conversation, separate from the first one: (a) First message: The same system prompt (figure 6) + class match prompt (figure 8) (b) Second message, same conversation, after GPT-4o replies: scoring prompt (figure 9) 13 Level Group Task Classes, videos per problem Ba...

  27. [36]

    Player holds an oak fence block in their hands

    We see a flat grassy area with small houses and trees nearby. Player holds an oak fence block in their hands

  28. [37]

    Same scene from a sligthly different angle

  29. [38]

    Player breaks a tuft of grass using an oak fence block

  30. [39]

    Player proceeds removing grass

  31. [40]

    Player places a fence block next to a house

  32. [41]

    Player places more fence blocks, building a straight fence line

  33. [42]

    Player turns around and finds a pig

  34. [43]

    Player places one more fence block in the line

  35. [44]

    Player breaks another tuft of grass

  36. [45]

    Figure 10: The prompt used to obtain frame-by-frame descriptions for Minecraft videos

    Player proceeds removing grass. Figure 10: The prompt used to obtain frame-by-frame descriptions for Minecraft videos. frames refer to the actual frames, which are base64 encoded and passed through the OpenAI API as detailed in their documentation. 17 Then, given the original ...

  37. [2023]

    URL http://arxiv.org/abs/2305.10683. 7

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.