{"id":"9960375b-d1bc-406e-8599-213653de77bd","arxiv_id":"2507.03578","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"General-purpose video foundation models, adapted with lightweight readout heads, reach state-of-the-art performance on three of five scientific video benchmarks.","lead":"The paper introduces SciVid, a benchmark of five science video tasks covering animal behavior, surgical tracking, and weather forecasting, and tests six off-the-shelf video AI models on them. It finds that general video models, adapted with simple readout heads, beat specialized models on three of the five tasks, but still trail specialized weather and tracking models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Digital Typhoon SOTA claim rests on unequal splits: SciVid's 80/20/20 split provides 4x the training data and a different test set than the 20% random-split baseline it is compared against, so Table 4 does not isolate ViFM transfer.","rationale":"The reader's weakest_assumption already identifies comparison fairness as the key issue, and I agree that the Digital Typhoon split mismatch is the most consequential instance. The central claim (abstract; Sec. 5.2) is that ViFMs achieve state-of-the-art results in several science tasks; Table 4 is the only SOTA result in a non-classification, non-animal-behavior domain. Because the training split differs (80/20/20 versus 20% random seeds) and the test sets are not the same, the comparison does not isolate the contribution of the frozen ViFM representation. The paper itself discloses the mismatch in Sec. 3.5 but does not address its effect. A matched-split rerun is straightforward given the released code and would settle the issue. The WeatherBench private-baseline/resolution concern is real but does not affect a SOTA claim, so it is secondary. STIR is transparently not SOTA, and the FlyVsFly/CalMS21 comparisons appear to use the same splits as prior work, though training-step and readout differences are less severe. Therefore I recommend keeping the reader's CONDITIONAL verdict; no verdict change is needed. If the matched-split test fails, the paper should be revised to claim strong transfer performance rather than SOTA on Digital Typhoon and to soften 'several applications.'","tokens_in":26959,"tokens_out":8174,"duration_ms":92717,"concrete_test":"Use the released SciVid code to re-run the Digital Typhoon experiment under Kitamoto et al.'s original protocol: train the same frozen 4DS-L readout on a 20% random sequence split (5 seeds), evaluate on that protocol's test sequences, and report average RMSE over steps {1,2,3,6,12}. If the matched-split RMSE remains near 5, the SOTA claim is confirmed; if it rises toward 10 or above, Table 4's comparison is invalid. Also report per-time-step RMSE and the number of test sequences so the averaging and test sets are verified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the Digital Typhoon SOTA comparison in Table 4. Section 3.5 states: 'While the results in the original paper [29] are computed for 20% train/test sequence splits for 5 random seeds, we generate a fixed 80%/20%/20% train/validation/test split.' Table 4 then lists Kitamoto et al.'s 11.71 test RMSE next to SciVid's 5.23. The two numbers are not obtained on the same test set, and the baseline had only one quarter as many training sequences. The gap could therefore reflect data quantity and test-set composition rather than the transferability of frozen video representations. Digital Typhoon is the only claimed SOTA result outside animal-behavior classification; if this comparison is invalid, the abstract's 'state-of-the-art results in several applications' loses its cross-domain support. WeatherBench 2 has a related transparency issue (Supp. Sec. A: privately communicated baselines at 0.25 degrees vs 1 degree), but since the paper does not claim SOTA there, it is secondary to this concern.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SciVid, a benchmark of five scientific video tasks (FlyVsFly, CalMS21, STIR, WeatherBench 2, Digital Typhoon) spanning animal behavior, medical imaging, and weather forecasting. Six video and image foundation models are evaluated by training lightweight task readouts on top of frozen backbones, with additional finetuning experiments for the two tasks where the frozen-backbone results lag. The authors report state-of-the-art results on CalMS21, FlyVsFly, and Digital Typhoon, non-trivial but below-SOTA results on STIR and WeatherBench 2, and analyze factors such as backbone scaling, readout architecture, temporal modeling, and training length. The code is released.","tokens_in":27253,"tokens_out":11507,"duration_ms":123498,"significance":"If its claims hold, SciVid would be a useful community asset: it offers a unified, explicitly documented evaluation protocol across diverse scientific domains, reports seed-to-seed noise, and provides evidence that generic video representations can transfer to out-of-domain scientific tasks. The frozen-backbone/readout design is simple and reproducible, and the code release is a concrete strength. The two animal-behavior SOTA claims are credible and well placed in context. However, the cross-domain SOTA conclusion rests on the Digital Typhoon comparison, which is currently not apples-to-apples, and the CalMS21 comparison lacks an explicit statement of the baseline's evaluation protocol. These issues are fixable but currently weaken the paper's main 'several applications' claim.","major_comments":[{"comment":"The Digital Typhoon state-of-the-art comparison is not conducted under equivalent conditions. Section 3.5 states that the original paper's numbers were computed on 20% train/test sequence splits over five random seeds, whereas SciVid uses a fixed 80/20/20 split. Table 4 reports Kitamoto et al.'s 11.71 test RMSE alongside SciVid's 5.23, but the two values are obtained on different test sequences, with SciVid using roughly four times more training data. The observed gap may therefore reflect data quantity and test-set composition rather than transfer from frozen video representations. To keep the 'state-of-the-art results in several applications' claim, the authors should evaluate Kitamoto et al.'s model on the SciVid split (or otherwise provide an equivalent training-data control), or restrict the claim to the two animal-behavior tasks.","section":"Sec. 3.5, Table 4"},{"comment":"The CalMS21 SOTA claim is not fully verifiable because the paper does not state the test split and preprocessing used for the VideoPrism-g baseline. Section 3.2 specifies that FlyVsFly uses the same splits as VideoPrism [70], but the CalMS21 paragraph describes only the authors' own subsampling (stride 16) and downscaling of the training set. Table 2 then compares SciVid's 92.0 mAP with VideoPrism's 91.5 mAP. If the VideoPrism number was produced on a different test set or at a different spatial resolution, the comparison is invalid; the manuscript should state explicitly which split and resolution were used for both methods.","section":"Sec. 3.2, Table 2"},{"comment":"Table 5 and Table C.6 are internally inconsistent as printed. For the 4DS-e frozen-backbone setting, Table 5 reports CalMS21 mAP 0.817 and FlyVsFly mAP 0.894, while Table 2 reports 92.0 and 84.6 (or 92.5 after 400k steps) for the corresponding entries; STIR accuracy is 0.513 in Table 5 but Table C.6 lists a mean of 0.746 for the 'same setting as Tab. 5.' The table captions do not state whether these are validation or test results, or whether the mAP values are percentages or fractions. Please specify the evaluation split and units, and reconcile the numbers, so that the backbone-comparison table can be used as the paper's central evidence.","section":"Table 5, Table C.6"}],"minor_comments":[{"comment":"Table 4's caption and the surrounding text refer to a 4DS-e backbone, but the table rows are labeled 4DS-L; please make the naming consistent.","section":"Table 4, Sec. 5.2"},{"comment":"The FlyVsFly SOTA row uses 400k training steps versus roughly 100k for the VideoPrism baseline; please add a matched-budget comparison or explicitly discuss the additional compute.","section":"Table 2, Sec. 5.2"},{"comment":"The GraphCast/GenCast/HRES baselines are evaluated at 0.25 degrees while SciVid operates at 1 degree, and the baseline numbers were privately communicated. The appendix asserts the resolution difference has negligible effect but does not show the supporting data; please include the 0.25 vs 1 degree comparison or make the baseline evaluations reproducible.","section":"Supp. A, Fig. 4"},{"comment":"The statement that Digital Typhoon's temporal-shuffle degradation is within noise would be more informative if it reported the actual mean difference and the corresponding standard deviation from the noise study.","section":"Sec. 5.1, Sec. C.13"}],"recommendation":"major_revision","confidential_remarks":"The paper is a valuable benchmark contribution with a clean protocol and code release. The main editorial decision hinges on whether the Digital Typhoon comparison can be made equitable; if not, the SOTA claim should be removed from the abstract and Sec. 5.2. The CalMS21 protocol also needs a one-paragraph clarification. If the authors address these points, the paper would be a solid contribution to the video foundation model evaluation literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading and worth a serious referee, but the headline SOTA claim needs to be walked back in one place. SciVid is a real contribution: five scientific video tasks across three domains, six backbones, a single frozen-backbone/readout protocol, and released code. The authors also do the right things around evaluation — they report seed-to-seed noise, ablate readouts, include a parameter-free resize baseline, and are honest that STIR and WeatherBench2 remain below domain-specific SOTA. The FlyVsFly and CalMS21 comparisons are on the same splits as VideoPrism, and the improvements there are credible.\n\nThe load-bearing soft spot is Digital Typhoon. Section 3.5 says the original Kitamoto et al. numbers use 20% train/test splits with five random seeds, while SciVid uses a fixed 80/20/20 split. Table 4 then puts Kitamoto's 11.71 next to SciVid's 5.23 and calls it SOTA. That is a fourfold difference in training data and a different test set. The gap cannot be attributed to ViFM transfer. Since Digital Typhoon is the only claimed SOTA outside animal-behavior classification, the abstract's \"state-of-the-art results in several applications\" is overstated. The fix is straightforward: retrain or re-evaluate Kitamoto's method on the same split, or report SciVid on the 20% split, and re-frame the claim accordingly.\n\nThe WeatherBench2 comparison is a smaller transparency issue: GraphCast/GenCast/HRES numbers were privately communicated and evaluated at 0.25 degrees while SciVid runs at 1 degree. The supplement says the difference is negligible, but the evidence is not public. This is minor because the paper does not claim SOTA there, but it should be cleaned up before publication.\n\nThe central transfer finding — frozen general video representations plus simple readouts beat domain-specific models in some, not all, scientific tasks — holds up on the animal behavior side and is a useful data point. The benchmark itself is the durable part of the paper. I would send it to review with the expectation that the Digital Typhoon comparison gets fixed. As is, the paper is a conditional accept, not a reject.","headline":"Useful cross-domain benchmark with credible transfer results on animal behavior, but the Digital Typhoon SOTA claim rests on unequal splits and should be re-framed.","tokens_in":27795,"tokens_out":2812,"would_cite":true,"duration_ms":32800,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Frozen video foundation models, pretrained on ordinary footage, can be transferred to scientific tasks with simple readout modules, and on three of the five SciVid benchmarks they beat the domain-specific state of the art.","keywords":["video foundation models","cross-domain transfer","animal behavior classification","surgical tissue tracking","weather forecasting","typhoon intensity forecasting","benchmark suite","representation learning"],"falsifier":"Re-run the SciVid Digital Typhoon pipeline under the original published protocol, 20 percent training split, five random seeds, RMSE at time steps 1, 2, 3, 6 and 12, and compare against the published baseline on identical data; if the frozen-ViFM margin shrinks to the task's measured noise (a standard deviation of 0.113 on a mean of 4.32), the state-of-the-art claim on that benchmark is not established. A complementary check is to reproduce the WeatherBench 2 baselines on the same 1 degree evaluation grid that SciVid uses, since the GraphCast, GenCast, and HRES figures were evaluated at 0.25 degrees.","tokens_in":26777,"feed_emoji":"🎥","tokens_out":13144,"duration_ms":130464,"temperature":0.7,"pith_summary":"This paper tries to establish that video foundation models (ViFMs), spatiotemporal networks pretrained on large collections of ordinary videos, can be transferred to scientific tasks far outside their training distribution and can beat models built specifically for those domains. To test this, the authors introduce SciVid, a benchmark of five tasks spanning animal-behavior classification, surgical tissue tracking, and weather and cyclone forecasting, and they adapt six leading ViFMs to each task with small trainable readout modules on top of frozen features. On three of the five tasks (two animal-behavior benchmarks and Digital Typhoon pressure forecasting) the frozen-backbone recipe reaches state-of-the-art results, while on the other two it yields non-trivial baselines that improve substantially when the backbone is finetuned. If the claim holds, it matters because scientists could avoid building bespoke models for each new imaging modality and instead attach lightweight readouts to general-purpose video representations.","feed_headline":"Frozen video models beat bespoke science AI on 3 of 5 tasks","feed_subtitle":"Simple readouts on out-of-domain video features reach the best published scores in three science benchmarks.","key_machinery":"The mechanism carrying the argument is a two-part adaptation protocol: a frozen pretrained ViFM backbone that encodes spatiotemporal clips, plus a small task-specific readout trained from scratch on labeled scientific data. The readouts are kept deliberately simple, a cross-attention module with a single learned query for behavior classification and cyclone pressure regression, a cross-attention module with point-position queries for surgical tracking, and a Dense Prediction Transformer for dense weather forecasting, so that any performance advantage must come from the backbone's representations rather than from an elaborate task model. The evaluation also includes a parameter-free resize baseline to confirm that the pretrained features carry non-trivial signal beyond raw pixels.","core_discovery":"The paper's central claim is that video foundation models can attain state-of-the-art results in science-related tasks even when pretrained on out-of-domain data and adapted with simple trainable readout modules. Concretely: frozen features from V-JEPA-H reach 92.4 mAP on CalMS21, frozen VideoPrism-g reaches 92.5 mAP on FlyVsFly after longer readout training, and frozen 4DS-L features reach 5.23 average RMSE on the Digital Typhoon test set, well below the 11.71 reported by the method that introduced that dataset. On STIR and WeatherBench 2 the same recipe does not reach state of the art, and finetuning helps but leaves a gap: 4DS-e tracking accuracy rises from 57.8% to 69.2% on the STIR test set against the 77.6% of the leading tracker, while LoRA with 2.3% of backbone parameters approaches full finetuning. The paper also reports that no single backbone wins across all five tasks, that cross-attention readouts clearly outperform linear ones, and that temporal order is load-bearing, since shuffling input frames degrades performance on most tasks.","pith_inferences":["The results sketch a roadmap toward one shared scientific video encoder: if out-of-domain features transfer this well, pretraining jointly on scientific and natural video could lift all five tasks at once, an experiment the paper does not run.","The WeatherBench 2 shortfall marks a likely boundary: physics-governed forecasting seems to need more than generic video representations, and feeding ViFM features into a dynamics-aware or physically constrained decoder is a natural next test.","The Digital Typhoon margin should be re-verified under the original evaluation protocol before it is leaned on, because the split mismatch with the published baseline, together with the strong 'copy last pressure' oracle, suggests the ViFM advantage may concentrate at longer forecast horizons."],"forward_implications":["One frozen video encoder can serve very different scientific domains: training readouts for all five tasks takes under a day on a single H100 GPU, so the marginal cost of entering a new domain is small.","The better backbone keeps its edge in low-data regimes (it still leads at 1/256 of the CalMS21 training data), which points to frozen ViFM features as a practical starting point for data-scarce scientific applications.","Backbone choice is task-dependent: 4DS-e leads on tracking, V-JEPA-H on fly behavior, and pixel-space MAE-style models on weather, so per-task model selection is currently necessary.","On the two tasks where frozen features lag, full finetuning or LoRA (about 2.3 percent of backbone parameters) recovers much of the gap, meaning the frozen-feature numbers are a floor rather than a ceiling.","Temporal information is load-bearing: shuffling frames or repeating a single frame consistently degrades performance, which explains why image-only backbones underperform video models across the suite."],"supporting_citations":[{"why":"Supplies the frozen-backbone-plus-readout adaptation paradigm, the readout architectures reused throughout, and the 4DS backbone family used for most ablations.","marker":"[11]"},{"why":"Provides the previous state of the art on FlyVsFly and CalMS21 that the SciVid readouts must beat, along with the mAP protocol and data splits adopted for those tasks.","marker":"[70]"},{"why":"Introduces the Digital Typhoon dataset and the published intensity-forecasting baseline (11.71 RMSE) that the ViFM readout claims to outperform.","marker":"[29]"},{"why":"Sets the state of the art on STIR tracking (MFT) against which the 4DS-e readout is measured.","marker":"[41]"},{"why":"Defines the WeatherBench 2 benchmark, its data splits, and the area-weighted RMSE metric used for the weather task.","marker":"[48]"},{"why":"Provides the Dense Prediction Transformer used as the dense weather-forecasting readout.","marker":"[47]"},{"why":"Supplies the GraphCast baseline and the channel-weighted loss convention used when training the weather readout.","marker":"[32]"},{"why":"Provides the FlyVsFly video dataset and its expert social-behavior annotations.","marker":"[19]"},{"why":"Provides the CalMS21 mouse-behavior dataset and its Task 1 labels.","marker":"[55]"},{"why":"Provides the STIR surgical tissue tracking dataset and its evaluation convention.","marker":"[51]"}],"fun_headline_variants":["Frozen video features hit SOTA on 3 of 5 science tasks","SciVid shows video models can beat science-specific AI on 3 tasks","Out-of-domain video features excel on 3 science benchmarks","Can a video model do science? SciVid says yes for 3 of 5"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim of state-of-the-art results stands or falls on whether each domain-specific baseline was evaluated under the same conditions as the SciVid pipeline, and that equivalence is not complete: the Digital Typhoon baseline used a smaller 20 percent training split over five random seeds, and the WeatherBench 2 baseline numbers were produced at a different resolution and communicated privately.","fun_headline_variants_meta":{"raw":{"variants":["Frozen video features hit SOTA on 3 of 5 science tasks","SciVid shows video models can beat science-specific AI on 3 tasks","Out-of-domain video features excel on 3 science benchmarks","Can a video model do science? SciVid says yes for 3 of 5"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001859,"raw_usage":{"total_tokens":7349,"prompt_tokens":1041,"completion_tokens":6308,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":6238}},"tokens_in":657,"tokens_out":6308,"duration_ms":52297,"temperature":1.0,"reasoning_tokens":6238,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:07:12.000600+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the SciVid Digital Typhoon pipeline under the original published protocol, 20 percent training split, five random seeds, RMSE at time steps 1, 2, 3, 6 and 12, and compare against the published baseline on identical data; if the frozen-ViFM margin shrinks to the task's measured noise (a standard deviation of 0.113 on a mean of 4.32), the state-of-the-art claim on that benchmark is not established. A complementary check is to reproduce the WeatherBench 2 baselines on the same 1 degree evaluation grid that SciVid uses, since the GraphCast, GenCast, and HRES figures were evaluated at 0.25 degrees.","supporting_citations":[{"cited_title":"Videoprism: A foundational visual encoder for video understanding","cited_arxiv_id":null,"evidence_quote":"Provides the previous state of the art on FlyVsFly and CalMS21 that the SciVid readouts must beat, along with the mAP protocol and data splits adopted for those tasks."},{"cited_title":"Digital typhoon: Long-term satellite image dataset for the spatio-temporal modeling of tropical cyclones","cited_arxiv_id":null,"evidence_quote":"Introduces the Digital Typhoon dataset and the published intensity-forecasting baseline (11.71 RMSE) that the ViFM readout claims to outperform."},{"cited_title":"MFT: Long- term tracking of every pixel","cited_arxiv_id":null,"evidence_quote":"Sets the state of the art on STIR tracking (MFT) against which the 4DS-e readout is measured."},{"cited_title":"Weatherbench 2: A benchmark for the next generation of data-driven global weather models.Journal of Advances in Modeling Earth Systems, 2024","cited_arxiv_id":null,"evidence_quote":"Defines the WeatherBench 2 benchmark, its data splits, and the area-weighted RMSE metric used for the weather task."},{"cited_title":"Vi- sion transformers for dense prediction","cited_arxiv_id":null,"evidence_quote":"Provides the Dense Prediction Transformer used as the dense weather-forecasting readout."},{"cited_title":"Learning skillful medium-range global weather forecasting.Science, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the GraphCast baseline and the channel-weighted loss convention used when training the weather readout."},{"cited_title":"Detecting social actions of fruit flies","cited_arxiv_id":null,"evidence_quote":"Provides the FlyVsFly video dataset and its expert social-behavior annotations."},{"cited_title":"The multi-agent behavior dataset: Mouse dyadic social interac- tions","cited_arxiv_id":null,"evidence_quote":"Provides the CalMS21 mouse-behavior dataset and its Task 1 labels."},{"cited_title":"DiMaio, and Sep- timiu E","cited_arxiv_id":null,"evidence_quote":"Provides the STIR surgical tissue tracking dataset and its evaluation convention."}],"review_version":1}