Pith. sign in

REVIEW 10 cited by

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1906.03327 v2 pith:WBMFJVWS submitted 2019-06-07 cs.CV

Antoine Miech , Dimitri Zhukov , Jean-Baptiste Alayrac , Makarand Tapaswi , Ivan Laptev , Josef Sivic This is my paper
classification cs.CV
keywords datasetvideoclipsdatadatasetsembeddinghowto100mtext-video
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Learning text-video embeddings usually requires a dataset of video clips with manually provided captions. However, such datasets are expensive and time consuming to create and therefore difficult to obtain on a large scale. In this work, we propose instead to learn such embeddings from video data with readily available natural language annotations in the form of automatically transcribed narrations. The contributions of this work are three-fold. First, we introduce HowTo100M: a large-scale dataset of 136 million video clips sourced from 1.22M narrated instructional web videos depicting humans performing and describing over 23k different visual tasks. Our data collection procedure is fast, scalable and does not require any additional manual annotation. Second, we demonstrate that a text-video embedding trained on this data leads to state-of-the-art results for text-to-video retrieval and action localization on instructional video datasets such as YouCook2 or CrossTask. Finally, we show that this embedding transfers well to other domains: fine-tuning on generic Youtube videos (MSR-VTT dataset) and movies (LSMDC dataset) outperforms models trained on these datasets alone. Our dataset, code and models will be publicly available at: www.di.ens.fr/willow/research/howto100m/.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs

    cs.CV 2026-07 conditional novelty 7.0 of 10

    TimeLens2 shows that a compact video MLLM can localize multiple evidence intervals in long videos by training on verified interval labels and a Wasserstein-based time-distance reward.

  2. HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    Processed egocentric human video outperforms teleoperated real-robot trajectories as pretraining data for embodied foundation models, delivering 24% lower validation loss and 52.5-90% higher task success rates under m...

  3. The TIME Machine: On The Power of Motion for Efficient Perception

    cs.CV 2026-05 conditional novelty 7.0 of 10

    TIME, a masked autoencoder on synthetic point tracks, matches or exceeds large video models on temporal benchmarks with four orders of magnitude less training data and complements appearance features.

  4. TimeProVe: Propose, then Verify for Efficient Long Video Temporal Reasoning in Activities of Daily Living

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    TimeProVe proposes a propose-then-verify framework using lightweight action-based candidate evidence generation followed by targeted VLM verification for efficient long video temporal reasoning, achieving 7.3% improve...

  5. TeachObs: A Human-Validated Benchmark for Multimodal Teaching Observation and Model Evaluation

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    TeachObs is a new human-validated benchmark dataset and evaluation protocol for multimodal AI on classroom teaching observation, showing no model dominates across tracks and that models over-rate procedurally clear lessons.

  6. TeachObs: A Human-Validated Benchmark for Multimodal Teaching Observation and Model Evaluation

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    TeachObs provides human-validated segment- and lesson-level labels on 30 classroom videos and finds frontier VLMs inconsistent, frame-inflated, and over-generous versus expert raters.

  7. The TIME Machine: On The Power of Motion for Efficient Perception

    cs.CV 2026-05 conditional novelty 6.0 of 10

    A motion-only embedding trained on synthetic point tracks matches or beats large appearance-based video models on temporal tasks and improves them when combined.

  8. The TIME Machine: On The Power of Motion for Efficient Perception

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    TIME is a motion-based embedding from point tracks, trained only on synthetic data via masked autoencoding, that matches state-of-the-art video model performance with up to 10,000x less training data.

  9. HumanNet: Scaling Human-centric Video Learning to One Million Hours

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    HumanNet is a 1M-hour human-centric video dataset with interaction annotations that enables better vision-language-action model performance than equivalent robot data in a controlled test.

  10. World Action Models: The Next Frontier in Embodied AI

    cs.RO 2026-05 unverdicted novelty 4.0 of 10

    The paper introduces World Action Models as a new paradigm unifying predictive world modeling with action generation in embodied foundation models and provides a taxonomy of existing approaches.

Pith tools