Pith. sign in

REVIEW 3 major objections 4 minor 72 references

SciVid: Cross-Domain Evaluation of Video Models in Scientific Applications

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Frozen video foundation models, pretrained on ordinary footage, can be transferred to scientific tasks with simple readout modules, and on three of the five SciVid benchmarks they beat the domain-specific state of the art.

desk verdict Useful cross-domain benchmark with credible transfer results on animal behavior, but the Digital Typhoon SOTA claim rests on unequal splits and should be re-framed. read the letter →

arxiv 2507.03578 v1 pith:EKPGJ56M submitted 2025-07-04 cs.CV cs.AIcs.LG

Yana Hasson , Pauline Luc , Liliane Momeni , Maks Ovsjanikov , Guillaume Le Moing , Alina Kuznetsova , Ira Ktena , Jennifer J. Sun
show 5 more authors
Skanda Koppula Dilara Gokay Joseph Heyward Etienne Pot Andrew Zisserman
This is my paper
classification cs.CVcs.AIcs.LG
keywords videofoundationmodelscross-domaintransferanimalbehaviorclassificationsurgicaltissuetrackingweatherforecastingtyphoonintensitybenchmarksuiterepresentationlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that video foundation models (ViFMs), spatiotemporal networks pretrained on large collections of ordinary videos, can be transferred to scientific tasks far outside their training distribution and can beat models built specifically for those domains. To test this, the authors introduce SciVid, a benchmark of five tasks spanning animal-behavior classification, surgical tissue tracking, and weather and cyclone forecasting, and they adapt six leading ViFMs to each task with small trainable readout modules on top of frozen features. On three of the five tasks (two animal-behavior benchmarks and Digital Typhoon pressure forecasting) the frozen-backbone recipe reaches state-of-the-art results, while on the other two it yields non-trivial baselines that improve substantially when the backbone is finetuned. If the claim holds, it matters because scientists could avoid building bespoke models for each new imaging modality and instead attach lightweight readouts to general-purpose video representations.

What carries the argument

The mechanism carrying the argument is a two-part adaptation protocol: a frozen pretrained ViFM backbone that encodes spatiotemporal clips, plus a small task-specific readout trained from scratch on labeled scientific data. The readouts are kept deliberately simple, a cross-attention module with a single learned query for behavior classification and cyclone pressure regression, a cross-attention module with point-position queries for surgical tracking, and a Dense Prediction Transformer for dense weather forecasting, so that any performance advantage must come from the backbone's representations rather than from an elaborate task model. The evaluation also includes a parameter-free resize baseline to confirm that the pretrained features carry non-trivial signal beyond raw pixels.

What would settle it

Re-run the SciVid Digital Typhoon pipeline under the original published protocol, 20 percent training split, five random seeds, RMSE at time steps 1, 2, 3, 6 and 12, and compare against the published baseline on identical data; if the frozen-ViFM margin shrinks to the task's measured noise (a standard deviation of 0.113 on a mean of 4.32), the state-of-the-art claim on that benchmark is not established. A complementary check is to reproduce the WeatherBench 2 baselines on the same 1 degree evaluation grid that SciVid uses, since the GraphCast, GenCast, and HRES figures were evaluated at 0.25 degrees.

Watch

Extended reading notes

Core claim

The paper's central claim is that video foundation models can attain state-of-the-art results in science-related tasks even when pretrained on out-of-domain data and adapted with simple trainable readout modules. Concretely: frozen features from V-JEPA-H reach 92.4 mAP on CalMS21, frozen VideoPrism-g reaches 92.5 mAP on FlyVsFly after longer readout training, and frozen 4DS-L features reach 5.23 average RMSE on the Digital Typhoon test set, well below the 11.71 reported by the method that introduced that dataset. On STIR and WeatherBench 2 the same recipe does not reach state of the art, and finetuning helps but leaves a gap: 4DS-e tracking accuracy rises from 57.8% to 69.2% on the STIR test set against the 77.6% of the leading tracker, while LoRA with 2.3% of backbone parameters approaches full finetuning. The paper also reports that no single backbone wins across all five tasks, that cross-attention readouts clearly outperform linear ones, and that temporal order is load-bearing, since shuffling input frames degrades performance on most tasks.

Load-bearing premise

The claim of state-of-the-art results stands or falls on whether each domain-specific baseline was evaluated under the same conditions as the SciVid pipeline, and that equivalence is not complete: the Digital Typhoon baseline used a smaller 20 percent training split over five random seeds, and the WeatherBench 2 baseline numbers were produced at a different resolution and communicated privately.

Editorial extensions

If this is right

  • One frozen video encoder can serve very different scientific domains: training readouts for all five tasks takes under a day on a single H100 GPU, so the marginal cost of entering a new domain is small.
  • The better backbone keeps its edge in low-data regimes (it still leads at 1/256 of the CalMS21 training data), which points to frozen ViFM features as a practical starting point for data-scarce scientific applications.
  • Backbone choice is task-dependent: 4DS-e leads on tracking, V-JEPA-H on fly behavior, and pixel-space MAE-style models on weather, so per-task model selection is currently necessary.
  • On the two tasks where frozen features lag, full finetuning or LoRA (about 2.3 percent of backbone parameters) recovers much of the gap, meaning the frozen-feature numbers are a floor rather than a ceiling.
  • Temporal information is load-bearing: shuffling frames or repeating a single frame consistently degrades performance, which explains why image-only backbones underperform video models across the suite.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The results sketch a roadmap toward one shared scientific video encoder: if out-of-domain features transfer this well, pretraining jointly on scientific and natural video could lift all five tasks at once, an experiment the paper does not run.
  • The WeatherBench 2 shortfall marks a likely boundary: physics-governed forecasting seems to need more than generic video representations, and feeding ViFM features into a dynamics-aware or physically constrained decoder is a natural next test.
  • The Digital Typhoon margin should be re-verified under the original evaluation protocol before it is leaned on, because the split mismatch with the published baseline, together with the strong 'copy last pressure' oracle, suggests the ViFM advantage may concentrate at longer forecast horizons.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces SciVid, a benchmark of five scientific video tasks (FlyVsFly, CalMS21, STIR, WeatherBench 2, Digital Typhoon) spanning animal behavior, medical imaging, and weather forecasting. Six video and image foundation models are evaluated by training lightweight task readouts on top of frozen backbones, with additional finetuning experiments for the two tasks where the frozen-backbone results lag. The authors report state-of-the-art results on CalMS21, FlyVsFly, and Digital Typhoon, non-trivial but below-SOTA results on STIR and WeatherBench 2, and analyze factors such as backbone scaling, readout architecture, temporal modeling, and training length. The code is released.

Significance. If its claims hold, SciVid would be a useful community asset: it offers a unified, explicitly documented evaluation protocol across diverse scientific domains, reports seed-to-seed noise, and provides evidence that generic video representations can transfer to out-of-domain scientific tasks. The frozen-backbone/readout design is simple and reproducible, and the code release is a concrete strength. The two animal-behavior SOTA claims are credible and well placed in context. However, the cross-domain SOTA conclusion rests on the Digital Typhoon comparison, which is currently not apples-to-apples, and the CalMS21 comparison lacks an explicit statement of the baseline's evaluation protocol. These issues are fixable but currently weaken the paper's main 'several applications' claim.

major comments (3)
  1. [Sec. 3.5, Table 4] The Digital Typhoon state-of-the-art comparison is not conducted under equivalent conditions. Section 3.5 states that the original paper's numbers were computed on 20% train/test sequence splits over five random seeds, whereas SciVid uses a fixed 80/20/20 split. Table 4 reports Kitamoto et al.'s 11.71 test RMSE alongside SciVid's 5.23, but the two values are obtained on different test sequences, with SciVid using roughly four times more training data. The observed gap may therefore reflect data quantity and test-set composition rather than transfer from frozen video representations. To keep the 'state-of-the-art results in several applications' claim, the authors should evaluate Kitamoto et al.'s model on the SciVid split (or otherwise provide an equivalent training-data control), or restrict the claim to the two animal-behavior tasks.
  2. [Sec. 3.2, Table 2] The CalMS21 SOTA claim is not fully verifiable because the paper does not state the test split and preprocessing used for the VideoPrism-g baseline. Section 3.2 specifies that FlyVsFly uses the same splits as VideoPrism [70], but the CalMS21 paragraph describes only the authors' own subsampling (stride 16) and downscaling of the training set. Table 2 then compares SciVid's 92.0 mAP with VideoPrism's 91.5 mAP. If the VideoPrism number was produced on a different test set or at a different spatial resolution, the comparison is invalid; the manuscript should state explicitly which split and resolution were used for both methods.
  3. [Table 5, Table C.6] Table 5 and Table C.6 are internally inconsistent as printed. For the 4DS-e frozen-backbone setting, Table 5 reports CalMS21 mAP 0.817 and FlyVsFly mAP 0.894, while Table 2 reports 92.0 and 84.6 (or 92.5 after 400k steps) for the corresponding entries; STIR accuracy is 0.513 in Table 5 but Table C.6 lists a mean of 0.746 for the 'same setting as Tab. 5.' The table captions do not state whether these are validation or test results, or whether the mAP values are percentages or fractions. Please specify the evaluation split and units, and reconcile the numbers, so that the backbone-comparison table can be used as the paper's central evidence.
minor comments (4)
  1. [Table 4, Sec. 5.2] Table 4's caption and the surrounding text refer to a 4DS-e backbone, but the table rows are labeled 4DS-L; please make the naming consistent.
  2. [Table 2, Sec. 5.2] The FlyVsFly SOTA row uses 400k training steps versus roughly 100k for the VideoPrism baseline; please add a matched-budget comparison or explicitly discuss the additional compute.
  3. [Supp. A, Fig. 4] The GraphCast/GenCast/HRES baselines are evaluated at 0.25 degrees while SciVid operates at 1 degree, and the baseline numbers were privately communicated. The appendix asserts the resolution difference has negligible effect but does not show the supporting data; please include the 0.25 vs 1 degree comparison or make the baseline evaluations reproducible.
  4. [Sec. 5.1, Sec. C.13] The statement that Digital Typhoon's temporal-shuffle degradation is within noise would be more informative if it reported the actual mean difference and the corresponding standard deviation from the noise study.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SciVid is an empirical benchmark comparison; the SOTA claims are not derived from the baselines they compare against.

full rationale

The paper's central claim is empirical: frozen ViFM backbones plus lightweight readouts reach strong results on five scientific video tasks. The load-bearing inputs are independently pretrained backbones (VideoMAE, V-JEPA, DINOv2, VideoPrism) and task readouts trained from scratch on each dataset's supervision. No equation or protocol in the paper defines a predicted quantity in terms of the baseline numbers it is compared with, and no fitted parameter is renamed as a prediction. The reuse of readout architectures and the 4DS backbone from the same team's prior paper [11] is not circular: those components are pretrained or designed outside SciVid, and the headline results are also achieved with externally developed backbones such as V-JEPA-H and DINOv2-L, so the central claim does not reduce to a self-citation. The Digital Typhoon comparison in Table 4 does involve a protocol mismatch (Sec. 3.5 states that the original Kitamoto et al. numbers use a 20% train/test split while SciVid uses a fixed 80/20/20 split), and the WeatherBench 2 baselines in Supp. Sec. A were privately communicated and evaluated at 0.25 degrees rather than the 1 degree used for SciVid. These are legitimate concerns about comparison fairness, reproducibility, and the precision of the 'state-of-the-art' wording, but they are not circularity: the SciVid RMSE values are produced by the paper's own training and evaluation pipeline, independent of the baseline numbers, and are not mathematically forced by the baseline definitions. The paper also includes control baselines, ablations, noise measurements, and a shuffle test, further separating its measurements from its claims. No self-definitional loop, fitted-input-as-prediction step, or load-bearing self-citation chain is present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The benchmark introduces no new physical or theoretical entities. Its central claims rest on a set of domain assumptions about dataset ground truth, a synthetic-to-real transfer assumption for the tracking readout, and a hand-chosen set of hyperparameters and data splits, several of which undermine the cleanest SOTA comparisons.

free parameters (5)
  • Backbone learning rate multiplier = 0.01 (STIR), 1 (WeatherBench 2)
    Chosen per task to maximize validation performance; the authors note that the optimal value for one task harms the other, so it is a task-specific hand-tuned choice that materially affects finetuning results in Fig. C.1.
  • Readout training steps = 40k (default), 10k (WeatherBench 2)
    Selected as a compromise based on validation; WeatherBench 2 overfits after 10k, while 80k helps some tasks, so this choice affects reported numbers.
  • Digital Typhoon train/val/test split = 80/20/20 fixed
    The authors replace the original 20% train split with five seeds by a fixed 80/20/20 split; this changes training data volume and makes the SOTA comparison non-equivalent.
  • Digital Typhoon pressure offset = 983.9 hPa
    Mean train pressure used as a regression offset; it is a train-set statistic, not a learned weight, but it is a hand-injected prior into the output space.
  • CalMS21 training subsampling = temporal stride 16, 2x spatial downscale
    Training data are subsampled to reduce memory; the paper claims this still surpasses SOTA, but it changes the training distribution relative to baselines that used the full data.
assumptions (5)
  • domain assumption ERA5 reanalysis is treated as ground truth for weather forecasting.
    Sec. 3.4 states WeatherBench 2 consists of reanalysis data, a best guess of the atmospheric state; all wRMSE numbers inherit this ground truth.
  • domain assumption Infrared tattoo segmentation gives correct STIR tracking targets.
    Sec. 3.3: tracked locations are automatically extracted by segmenting infrared tattoos in initial and final frames; errors in segmentation propagate to all tracking accuracy scores.
  • domain assumption Closest-target heuristic matching provides a valid STIR accuracy estimate.
    Supp. Sec. B.2 states no query-target matching is available, so a closest-target point heuristic is used; the computed accuracy is an estimate and can include erroneous matchings.
  • domain assumption Kubric synthetic data is a sufficient pretraining distribution for the tracking readout.
    Supp. Sec. B.1: the STIR readout is trained on 16-frame windows from the Kubric MOVi-E dataset; transfer to surgical tissue is assumed, not proven.
  • ad hoc to paper A 1 degree grid is close enough to ViFM pretraining resolution and to 0.25 degree SOTA baselines.
    Sec. 3.4 chooses 1 degree for ViFM compatibility; Supp. Sec. A asserts private baseline results at 1 degree are indistinguishable, without showing a reproducible evaluation protocol.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SciVid: Cross-Domain Evaluation of Video Models in Scientific Applications." pith.science (2026). https://pith.science/paper/EKPGJ56M

@misc{pith2026250703578,
  author       = {Pith},
  title        = {Pith review of: SciVid: Cross-Domain Evaluation of Video Models in Scientific Applications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EKPGJ56M}},
  note         = {Machine review of arXiv:2507.03578}
}
read the original abstract

In recent years, there has been a proliferation of spatiotemporal foundation models in different scientific disciplines. While promising, these models are often domain-specific and are only assessed within the particular applications for which they are designed. Given that many tasks can be represented as video modeling problems, video foundation models (ViFMs) hold considerable promise as general-purpose domain-agnostic approaches. However, it is not known whether the knowledge acquired on large-scale but potentially out-of-domain data can be effectively transferred across diverse scientific disciplines, and if a single, pretrained ViFM can be competitive with domain-specific baselines. To address this, we introduce SciVid, a comprehensive benchmark comprising five *Sci*entific *Vid*eo tasks, across medical computer vision, animal behavior, and weather forecasting. We adapt six leading ViFMs to SciVid using simple trainable readout modules, establishing strong baselines and demonstrating the potential for effective transfer learning. Specifically, we show that state-of-the-art results can be obtained in several applications by leveraging the general-purpose representations from ViFM backbones. Furthermore, our results reveal the limitations of existing ViFMs, and highlight opportunities for the development of generalizable models for high-impact scientific applications. We release our code at https://github.com/google-deepmind/scivid to facilitate further research in the development of ViFMs.

Figures

Figures reproduced from arXiv: 2507.03578 by the authors.

Figure 1
Figure 1. SCIVID: A diverse collection of five scientific video tasks – sourced from animal behavior, medical imaging and weather fore￾casting domains – that evaluates a range of spatio-temporal reasoning capabilities, including classification, point tracking and forecasting. We show for each task, inputs (yellow), ground truth (blue) and predictions (pink) from our framework with the 4DS-e [11] backbone. We stress that SCIVI… view at source ↗
Figure 2
Figure 2. ). In this section, we provide the details on the vision foundation backbones (Sec. 4.1) and readout architectures (Sec. 4.2). Our experimental setup with frozen backbones requires less than a day to complete on a single H100 GPU, for all ViFMs and all five SCIVID tasks. Further implemen￾tation details can be found in Sec. B of the supp. mat [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Temporal modeling. We show the (percentage) per￾formance degradation with respect to our readout training with the frozen 4DS-e backbone ( ) and using ordered video frames, com￾pared to random shuffling. For STIR we ensure that the tracked points correctly correspond to each frame after shuffling. Distribution shift. We further provide a study of the shift between the video distributions of typical ViFM pretraining … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: SOTA Comparison Weatherbench 2. Our readout on top of frozen ( ) 4DS-e and VideoMAEv2-g backbones obtains non-trivial, yet modest performance compared with state-of-the￾art approaches (HRES, GraphCast and GenCast). Finetuning the backbones ( ) helps significantly but a…
Figure 5
Figure 5. Figure 5: Backbone scaling. We compare performance for smaller and larger variants of models from different model families, when training readouts on top of frozen backbones ( ). While larger variants from a same family tend to perform better on FlyVsFly and CalMS21, we observe …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

72 extracted references · 45 canonical work pages

  1. [70]

    Videoprism: A foundational visual encoder for video understanding

    Long Zhao, Nitesh Bharadwaj Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, et al. Videoprism: A foundational visual encoder for video understanding. In ICML, 2024. 2, 3, 4, 6

    work page 2024

  2. [1]

    Ac- cessed: November 7, 2024

    Stir challenge 2024.https://www.synapse.org/ Synapse:syn54126082/wiki/626617, 2024. Ac- cessed: November 7, 2024. 3, 6, 2

    work page 2024

  3. [2]

    Flamingo: a visual language model for few-shot learning.NeurIPS, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.NeurIPS, 2022. 2

    2022

  4. [3]

    Vivit: A video vi- sion transformer

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lu ˇci´c, and Cordelia Schmid. Vivit: A video vi- sion transformer. InICCV, 2021. 4

    work page 2021

  5. [4]

    Foundation models defining a new era in vision: a survey and outlook

    Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Foundation models defining a new era in vision: a survey and outlook. PAMI, 2025. 2

    work page 2025

  6. [5]

    Evaluating general purpose vision foundation models for medical im- age analysis: An experimental study of dinov2 on radiology benchmarks.arXiv preprint arXiv:2312.02366, 2023

    Mohammed Baharoon, Waseem Qureshi, Jiahong Ouyang, Yanwu Xu, Abdulrhman Aljouie, and Wei Peng. Evaluating general purpose vision foundation models for medical im- age analysis: An experimental study of dinov2 on radiology benchmarks.arXiv preprint arXiv:2312.02366, 2023. 1

    arXiv 2023

  7. [6]

    Automated audiovisual be- havior recognition in wild primates.Science Advances, 2021

    Max Bain, Arsha Nagrani, Daniel Schofield, Sophie Berdugo, Joana Bessa, Jake Owen, Kimberley Hockings, Tetsuro Matsuzawa, Misato Hayashi, Dora Biro, Susana Car- valho, and Andrew Zisserman. Automated audiovisual be- havior recognition in wild primates.Science Advances, 2021. 1

    work page 2021

  8. [7]

    Revisiting feature prediction for learning visual rep- resentations from video.TMLR, 2024

    Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual rep- resentations from video.TMLR, 2024. 2, 4, 6

    work page 2024

Show all 72 references
  1. [8]

    Accurate medium-range global weather forecasting with 3d neural networks.Nature, 2023

    Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiao- tao Gu, and Qi Tian. Accurate medium-range global weather forecasting with 3d neural networks.Nature, 2023. 4

    2023

  2. [9]

    Bruinsma, Ana Lucic, Megan Stanley, Anna Vaughan, Johannes Brandstetter, Patrick Gar- van, Maik Riechert, Jonathan A

    Cristian Bodnar, Wessel P. Bruinsma, Ana Lucic, Megan Stanley, Anna Vaughan, Johannes Brandstetter, Patrick Gar- van, Maik Riechert, Jonathan A. Weyn, Haiyu Dong, Jayesh K. Gupta, Kit Thambiratnam, Alexander T. Archibald, Chun-Chieh Wu, Elizabeth Heider, Max Welling, Richard E...

    2024 arXiv

  3. [10]

    Hudson, Ehsan Adeli, Russ Alt- man, Simran Arora, Sydney von Arx, Michael S

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Alt- man, Simran Arora, Sydney von Arx, Michael S. Bern- stein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castel- lon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. C...

    2021 arXiv

  4. [11]

    Jo ˜ao Carreira, Dilara Gokay, Michael King, Chuhan Zhang, Ignacio Rocco, Aravindh Mahendran, Thomas Albert Keck, Joseph Heyward, Skanda Koppula, Etienne Pot, Goker Erdo- gan, Yana Hasson, Yi Yang, Klaus Greff, Guillaume Le Mo- ing, Sjoerd van Steenkiste, Daniel Zoran, Drew A....

    2024 arXiv

  5. [12]

    Physical understanding of the tropical cyclone wind-pressure relation- ship.Nature Communications, 2017

    Daniel R Chavas, Kevin A Reed, and John A Knaff. Physical understanding of the tropical cyclone wind-pressure relation- ship.Nature Communications, 2017. 4 9

    2017

  6. [13]

    Fuxi: A cascade ma- chine learning forecasting system for 15-day global weather forecast.npj Climate and Atmospheric Science, 2023

    Lei Chen, Xiaohui Zhong, Feng Zhang, Yuan Cheng, Yinghui Xu, Yuan Qi, and Hao Li. Fuxi: A cascade ma- chine learning forecasting system for 15-day global weather forecast.npj Climate and Atmospheric Science, 2023. 4

    2023

  7. [14]

    Foundation models for weather and climate data understanding: A comprehensive survey.arXiv preprint arXiv:2312.03014, 2023

    Shengchao Chen, Guodong Long, Jing Jiang, Dikai Liu, and Chengqi Zhang. Foundation models for weather and climate data understanding: A comprehensive survey.arXiv preprint arXiv:2312.03014, 2023. 2

    2023 arXiv

  8. [15]

    Vision–language foundation model for echocardiogram interpretation.Nature Medicine, 2024

    Matthew Christensen, Milos Vukadinovic, Neal Yuan, and David Ouyang. Vision–language foundation model for echocardiogram interpretation.Nature Medicine, 2024. 1, 2

    2024

  9. [16]

    TAP-vid: A bench- mark for tracking any point in a video

    Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Re- casens Continente, Lucas Smaira, Yusuf Aytar, Joao Car- reira, Andrew Zisserman, and Yi Yang. TAP-vid: A bench- mark for tracking any point in a video. InNeurIPS, 2022. 3

    2022

  10. [17]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  11. [18]

    Scaling spherical cnns.arXiv preprint arXiv:2306.05420,

    Carlos Esteves, Jean-Jacques Slotine, and Ameesh Makadia. Scaling spherical cnns.arXiv preprint arXiv:2306.05420,

    arXiv

  12. [19]

    Detecting social actions of fruit flies

    Eyrun Eyjolfsdottir, Steve Branson, Xavier P Burgos- Artizzu, Eric D Hoopfer, Jonathan Schor, David J Anderson, and Pietro Perona. Detecting social actions of fruit flies. In ECCV, 2014. 3

    2014

  13. [20]

    PhilEO bench: Evaluating geo-spatial foundation models

    Casper Fibaek, Luke Camilleri, Andreas Luyts, Nikolaos Dionelis, and Bertrand Le Saux. PhilEO bench: Evaluating geo-spatial foundation models. InInternational Geoscience and Remote Sensing Symposium, 2024. 2

    2024

  14. [21]

    Spatio-temporal foundation models: Vision, challenges, and opportunities.arXiv preprint arXiv:2501.09045, 2025

    Adam Goodge, Wee Siong Ng, Bryan Hooi, and See Kiong Ng. Spatio-temporal foundation models: Vision, challenges, and opportunities.arXiv preprint arXiv:2501.09045, 2025. 2

    2025 arXiv

  15. [22]

    Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J. Fleet, Dan Gnanapra- gasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh- Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrou...

    2022

  16. [23]

    Self-supervised learning for en- doscopic video analysis

    Roy Hirsch, Mathilde Caron, Regev Cohen, Amir Livne, Ron Shapiro, Tomer Golany, Roman Goldenberg, Daniel Freedman, and Ehud Rivlin. Self-supervised learning for en- doscopic video analysis. InMICCAI, 2023. 1

    2023

  17. [24]

    LoRA: Low-rank adaptation of large lan- guage models

    Edward J Hu et al. LoRA: Low-rank adaptation of large lan- guage models. InICLR, 2022. 3, 4

    2022

  18. [25]

    Foundation models for spatiotemporal tasks in the physical world

    Zhe Jiang, Yu Wang, and Zelin Xu. Foundation models for spatiotemporal tasks in the physical world. InSIAM Interna- tional Conference on Data Mining, 2024. 2

    2024

  19. [26]

    Animal-bench: Benchmarking multimodal video models for animal-centric video understanding.NeuRIPS, 2024

    Yinuo Jing, Ruxu Zhang, Kongming Liang, Yongxiang Li, Zhongjiang He, Zhanyu Ma, and Jun Guo. Animal-bench: Benchmarking multimodal video models for animal-centric video understanding.NeuRIPS, 2024. 2

    2024

  20. [27]

    The kinetics human action video dataset.arXiv preprint arXiv:1705.06950, 2017

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset.arXiv preprint arXiv:1705.06950, 2017. 8, 14

    2017 arXiv

  21. [28]

    Forecasting global weather with graph neural networks.arXiv preprint arXiv:2202.07575, 2022

    Ryan Keisler. Forecasting global weather with graph neural networks.arXiv preprint arXiv:2202.07575, 2022. 4

    2022 arXiv

  22. [29]

    Digital typhoon: Long-term satellite image dataset for the spatio-temporal modeling of tropical cyclones

    Asanobu Kitamoto, Jared Hwang, Bastien Vuillod, Lucas Gautier, Yingtao Tian, and Tarin Clanuwat. Digital typhoon: Long-term satellite image dataset for the spatio-temporal modeling of tropical cyclones. InNeurIPS, 2023. 3, 4, 6, 2, 5, 7, 8

    2023

  23. [30]

    Neural general circulation models for weather and climate.Nature, 2024

    Dmitrii Kochkov, Janni Yuval, Ian Langmore, Peter Nor- gaard, Jamie Smith, Griffin Mooers, Milan Kl ¨ower, James Lottes, Stephan Rasp, Peter D ¨uben, et al. Neural general circulation models for weather and climate.Nature, 2024. 4

    2024

  24. [31]

    Four- castnet: Accelerating global high-resolution weather fore- casting using adaptive fourier neural operators

    Thorsten Kurth, Shashank Subramanian, Peter Harring- ton, Jaideep Pathak, Morteza Mardani, David Hall, Andrea Miele, Karthik Kashinath, and Anima Anandkumar. Four- castnet: Accelerating global high-resolution weather fore- casting using adaptive fourier neural operators. InPro...

    2023

  25. [32]

    Learning skillful medium-range global weather forecasting.Science, 2023

    Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, Alexander Merose, Stephan Hoyer, George Holland, Orial Vinyals, Jacklynn Stott, Alexander Pritzel, Shakir Mohamed, and P...

    2023

  26. [33]

    VideoEval: Comprehensive benchmark suite for low-cost evaluation of video foundation model.arXiv preprint arXiv:2407.06491, 2024

    Xinhao Li, Zhenpeng Huang, Jing Wang, Kunchang Li, and Limin Wang. VideoEval: Comprehensive benchmark suite for low-cost evaluation of video foundation model.arXiv preprint arXiv:2407.06491, 2024. 2

    2024 arXiv

  27. [34]

    Video-llava: Learning united visual rep- resentation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual rep- resentation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023. 2

    2023 arXiv

  28. [35]

    Discriminative correlation filter tracker with channel and spatial reliability.IJCV, 2018

    Alan Luke ˇziˇc, Tom’a ˇs V oj’iˇr, Luka ˇCehovin Zajc, Ji ˇr’i Matas, and Matej Kristan. Discriminative correlation filter tracker with channel and spatial reliability.IJCV, 2018. 6, 5

    2018

  29. [36]

    Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023. 2

    2023 arXiv

  30. [37]

    Foundation mod- els for video understanding: A survey.arXiv preprint arXiv:2405.03770, 2024

    Neelu Madan, Andreas Møgelmose, Rajat Modi, Yogesh S Rawat, and Thomas B Moeslund. Foundation mod- els for video understanding: A survey.arXiv preprint arXiv:2405.03770, 2024. 1, 2

    2024 arXiv

  31. [38]

    Martinez

    Andrew B. Martinez. Forecast accuracy matters for hurri- cane damage.Econometrics, 2020. 4

    2020

  32. [39]

    Ai foundation models for weather and climate: Applications, design, and implementation.arXiv preprint arXiv:2309.10808, 2023

    S Karthik Mukkavilli, Daniel Salles Civitarese, Johannes Schmude, Johannes Jakubik, Anne Jones, Nam Nguyen, 10 Christopher Phillips, Sujit Roy, Shraddha Singh, Campbell Watson, et al. Ai foundation models for weather and climate: Applications, design, and implementation.arXiv ...

    2023 arXiv

  33. [40]

    Benchmarking foundation models as feature extractors for weakly-supervised computational pathology

    Peter Neidlinger, Omar SM El Nahhas, Hannah Sophie Muti, Tim Lenz, Michael Hoffmeister, Hermann Brenner, Marko van Treeck, Rupert Langer, Bastian Dislich, Hans Michael Behrens, et al. Benchmarking foundation models as feature extractors for weakly-supervised computational path...

    2024 arXiv

  34. [41]

    MFT: Long- term tracking of every pixel

    Michal Neoral, Jon ´aˇs ˇSer`ych, and Ji ˇr´ı Matas. MFT: Long- term tracking of every pixel. InWACV, 2024. 6, 5

    2024

  35. [42]

    Foundation mod- els for science: Progress, opportunities, and challenges (fm4science)

    NeurIPS 2024 FM4Science Workshop. Foundation mod- els for science: Progress, opportunities, and challenges (fm4science). NeurIPS 2024 Workshop, 2024. 2

    2024

  36. [43]

    Climax: A foun- dation model for weather and climate.arXiv preprint arXiv:2301.10343, 2023

    Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. Climax: A foun- dation model for weather and climate.arXiv preprint arXiv:2301.10343, 2023. 2

    2023 arXiv

  37. [44]

    Maxime Oquab, Timoth ´ee Darcet, Th´eo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael ...

    2024

  38. [45]

    Probabilistic weather forecasting with machine learn- ing.Nature, 2025

    Ilan Price, Alvaro Sanchez-Gonzalez, Ferran Alet, Tom R Andersson, Andrew El-Kadi, Dominic Masters, Timo Ewalds, Jacklynn Stott, Shakir Mohamed, Peter Battaglia, et al. Probabilistic weather forecasting with machine learn- ing.Nature, 2025. 4, 1

    2025

  39. [46]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. InICML, 2021. 1

    2021

  40. [47]

    Vi- sion transformers for dense prediction

    Ren ´e Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vi- sion transformers for dense prediction. InICCV, 2021. 5, 2, 6

    2021

  41. [48]

    Weatherbench 2: A benchmark for the next generation of data-driven global weather models.Journal of Advances in Modeling Earth Systems, 2024

    Stephan Rasp, Stephan Hoyer, Alexander Merose, Ian Langmore, Peter Battaglia, Tyler Russell, Alvaro Sanchez- Gonzalez, Vivian Yang, Rob Carver, Shreya Agrawal, et al. Weatherbench 2: A benchmark for the next generation of data-driven global weather models.Journal of Advances i...

    2024

  42. [49]

    Donti, Lynn H

    David Rolnick, Priya L. Donti, Lynn H. Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, An- drew Slavin Ross, Nikola Milojevic-Dupont, Natasha Jaques, Anna Waldman-Brown, Alexandra Sasha Luccioni, Tegan Maharaj, Evan D. Sherwin, S. Karthik Mukkavilli, Konrad P. Kordin...

    2022

  43. [50]

    General surgery vision transformer: A video pre-trained foundation model for general surgery.arXiv preprint arXiv:2403.05949, 2024

    Samuel Schmidgall, Ji Woong Kim, Jeffrey Jopling, and Axel Krieger. General surgery vision transformer: A video pre-trained foundation model for general surgery.arXiv preprint arXiv:2403.05949, 2024. 1, 2

    2024 arXiv

  44. [51]

    DiMaio, and Sep- timiu E

    Adam Schmidt, Omid Mohareri, Simon P. DiMaio, and Sep- timiu E. Salcudean. Surgical tattoos in infrared: A dataset for quantifying tissue tracking and mapping.IEEE Transactions on Medical Imaging, 2024. 3

    2024

  45. [52]

    Deep learning and founda- tion models for weather prediction: A survey.arXiv preprint arXiv:2501.06907, 2025

    Jimeng Shi, Azam Shirali, Bowen Jin, Sizhe Zhou, Wei Hu, Rahuul Rangaraj, Shaowen Wang, Jiawei Han, Zhao- nan Wang, Upmanu Lall, et al. Deep learning and founda- tion models for weather prediction: A survey.arXiv preprint arXiv:2501.06907, 2025. 2

    2025 arXiv

  46. [53]

    Us billion-dollar weather and climate disasters: Data sources, trends, accuracy and bi- ases.Natural Hazards, 2013

    Adam Smith and Richard Katz. Us billion-dollar weather and climate disasters: Data sources, trends, accuracy and bi- ases.Natural Hazards, 2013. 4

    2013

  47. [54]

    Toward collaborative artificial intelligence development for animal well-being.Journal of the American Veterinary Medical Association, 2025

    Jennifer J Sun. Toward collaborative artificial intelligence development for animal well-being.Journal of the American Veterinary Medical Association, 2025. 2

    2025

  48. [55]

    The multi-agent behavior dataset: Mouse dyadic social interac- tions

    Jennifer J Sun, Tomomi Karigo, Dipam Chakraborty, Sharada P Mohanty, Benjamin Wild, Quan Sun, Chen Chen, David J Anderson, Pietro Perona, Yisong Yue, et al. The multi-agent behavior dataset: Mouse dyadic social interac- tions. InNeurIPS, 2021. 3, 7

    2021

  49. [56]

    Sun, Ann Kennedy, Eric Zhan, David J

    Jennifer J. Sun, Ann Kennedy, Eric Zhan, David J. Anderson, Yisong Yue, and Pietro Perona. Task programming: Learn- ing data efficient behavior representations. InCVPR, 2021. 3

    2021

  50. [57]

    Video foundation models for animal behavior analysis.bioRxiv, 2024

    Jennifer J Sun, Hao Zhou, Long Zhao, Liangzhe Yuan, Bryan Seybold, David Hendon, Florian Schroff, David A Ross, Hartwig Adam, Bo Hu, et al. Video foundation models for animal behavior analysis.bioRxiv, 2024. 1, 2

    2024

  51. [58]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2024

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2024. 2

    2024 arXiv

  52. [59]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. InECCV, 2020. 6, 5

    2020

  53. [60]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.NeurIPS, 2022

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.NeurIPS, 2022. 2, 4, 8

    2022

  54. [61]

    Bergen, Carla P

    Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Pe- ter Van Katwyk, Andreea Deac, Anima Anandkumar, Kari- anne J. Bergen, Carla P. Gomes, Shirley Ho, Pushmeet Kohli, Joan Lasenby, Jure Leskovec, Tie-Yan Liu, Arjun K. Manr...

  55. [62]

    Videomae v2: Scaling video masked autoencoders with dual masking

    Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yi- nan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In CVPR, 2023. 2, 4, 3

    2023

  56. [63]

    Internvideo: General video foundation models via generative and discriminative learning.arXiv preprint arXiv:2212.03191, 2022

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun 11 Wang, et al. Internvideo: General video foundation models via generative and discriminative learning.arXiv preprint arXiv:2212.03191, 2022. 2

    2022 arXiv

  57. [64]

    Foun- dation model for endoscopy video analysis via large-scale self-supervised pre-train

    Zhao Wang, Chang Liu, Shaoting Zhang, and Qi Dou. Foun- dation model for endoscopy video analysis via large-scale self-supervised pre-train. InInternational Conference on Medical Image Computing and Computer-Assisted Interven- tion. Springer, 2023. 1, 2

    2023

  58. [65]

    A general-purpose multi- modal foundation model for dermatology.arXiv preprint arXiv:2410.15038, 2024

    Siyuan Yan, Zhen Yu, Clare Primiero, Cristina Vico-Alonso, Zhonghua Wang, Litao Yang, Philipp Tschandl, Ming Hu, Gin Tan, Vincent Tang, et al. A general-purpose multi- modal foundation model for dermatology.arXiv preprint arXiv:2410.15038, 2024. 2

    arXiv

  59. [66]

    X- pose: Detecting any keypoints

    Jie Yang, Ailing Zeng, Ruimao Zhang, and Lei Zhang. X- pose: Detecting any keypoints. InECCV, 2024. 2

    2024

  60. [67]

    Superanimal pretrained pose estimation models for behavioral analysis.Nature communi- cations, 2024

    Shaokai Ye, Anastasiia Filippova, Jessy Lauer, Steffen Schneider, Maxime Vidal, Tian Qiu, Alexander Mathis, and Mackenzie Weygandt Mathis. Superanimal pretrained pose estimation models for behavioral analysis.Nature communi- cations, 2024. 2

    2024

  61. [68]

    Videoglue: Video general understanding evaluation of foundation mod- els.arXiv preprint arXiv:2307.03166, 2023

    Liangzhe Yuan, Nitesh Bharadwaj Gundavarapu, Long Zhao, Hao Zhou, Yin Cui, Lu Jiang, Xuan Yang, Menglin Jia, Tobias Weyand, Luke Friedman, et al. Videoglue: Video general understanding evaluation of foundation mod- els.arXiv preprint arXiv:2307.03166, 2023. 2

    2023 arXiv

  62. [69]

    A generalist vision–language foundation model for diverse biomedical tasks.Nature Medicine, 2024

    Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D Davi- son, Hui Ren, et al. A generalist vision–language foundation model for diverse biomedical tasks.Nature Medicine, 2024. 2

    2024

  63. [71]

    Biomed- parse: a biomedical foundation model for image pars- ing of everything everywhere all at once.arXiv preprint arXiv:2405.12971, 2024

    Theodore Zhao, Yu Gu, Jianwei Yang, Naoto Usuyama, Ho Hin Lee, Tristan Naumann, Jianfeng Gao, Angela Crab- tree, Jacob Abel, Christine Moung-Wen, et al. Biomed- parse: a biomedical foundation model for image pars- ing of everything everywhere all at once.arXiv preprint arXiv:2...

    2024 arXiv

  64. [72]

    circling

    Zihao Zhao, Yuxiao Liu, Han Wu, Mei Wang, Yonghao Li, Sheng Wang, Lin Teng, Disheng Liu, Zhiming Cui, Qian Wang, et al. Clip in medical imaging: A comprehensive sur- vey.arXiv preprint arXiv:2312.07353, 2023. 1 12 SCIVID: Cross-Domain Evaluation of Video Models in Scientific A...

    2018 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.