{"id":"f8e71993-bae1-4f92-8088-77c0352ad166","arxiv_id":"2412.18386","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The authors train a view-switch predictor on pseudo-labeled web videos and show it transfers to selecting ego/exo views in new multi-view videos with limited labels.","lead":"This paper introduces SWITCH-A-VIEW, a model that learns when to switch camera views in how-to videos by training on unlabeled, already-edited instructional videos. It could automate video editing decisions and help create better instructional content from multi-camera footage.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The pseudo-labeler's ego/exo accuracy on HowTo100M is never validated, and the HT100M test set oversamples its detected boundaries; the reported view-switch gains may partly reflect learning the pseudo-labeler's biases (e.g., scene-cut detection) rather than human view-switch preferences.","rationale":"The reader identified the pseudo-labeler's accuracy as the weakest assumption. My analysis agrees but sharpens it into a specific mechanism: the pseudo-labeler is trained on Charades-Ego, applied to in-the-wild HT100M, and its shot boundaries come from PySceneDetect. Because the model is trained to predict pseudo-labeled view switches that only occur at scene-cut boundaries, it may learn a shortcut based on visual discontinuity rather than human-editing intent. The HT100M test set's oversampling of clips right after pseudo-labeler-identified boundaries further risks selection bias toward exactly the instances the pseudo-labeler finds easy. These issues do not necessarily invalidate the approach—the Ego-Exo4D zero-shot and few-shot results provide independent, if modest, evidence of transfer—but they do mean the large HT100M numbers should be interpreted cautiously. The proposed checks would directly measure pseudo-label quality and the impact of the biased test sampling, which are the load-bearing assumptions behind the pretext-task claim. Since the reader's conditional verdict already requires addressing the pseudo-label concern, my read does not change the verdict; it only adds specificity about why that concern is central and how to resolve it.","tokens_in":21432,"tokens_out":9370,"duration_ms":89560,"concrete_test":"Compute the pseudo-labeler's shot-level ego/exo accuracy against human labels on the HT100M test set (3,151 instances). If accuracy is below roughly 80%, the training signal is substantially noisy. Additionally, re-evaluate view-switch detection on a uniformly sampled HT100M test subset without the Supp 7.9 boundary oversampling; if SWITCH-A-VIEW's AUC margin over Retrieval-F drops below about 5 points, the reported 10.4-point margin is partly an artifact of test-set selection. Also compare zero-shot Ego-Exo4D performance of a model pretrained on pseudo-labels versus the same model pretrained on human-labeled HT100M clips (if such labels can be obtained); if the clean-pretrained model does not exceed the pseudo-label model, the pseudo-label noise is not the limiting factor, weakening the concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the pseudo-labeler producing ego/exo labels that approximate human view-switch decisions in unlabeled how-to videos. This is never validated. Concretely: (1) The Charades-Ego-trained classifier is applied to in-the-wild HT100M with no reported accuracy against human labels; (2) shot-level pseudo-labels are defined by PySceneDetect boundaries, so the training task is effectively 'at a detected scene cut, predict whether the next shot's pseudo-labeled view differs'—the model may learn low-level cut detection rather than human editing preferences; (3) the HT100M test set is constructed by oversampling clips right after pseudo-labeler-detected boundaries (Supp 7.9), and evaluation uses the same pseudo-labeler for past views (Implementation), so the large HT100M margins (AUC 63.8 vs 53.4) may partly reflect agreement with the pseudo-labeler's biases. The independent Ego-Exo4D zero-shot result is more trustworthy but modest (AUC 56.4 vs 53.4), and the fine-tuned view-selection gains over LangView-bigData are about 2.5 AUC—consistent with a generic ego/exo-adapted representation rather than a learned model of human view-switch patterns. Without pseudo-label validation or an unbiased HT100M eval, the strength of the headline claim is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SWITCH-A-VIEW, a method for view selection in multi-camera instructional videos. It trains a view-switch detector on unlabeled HowTo100M videos using pseudo-labels: PySceneDetect boundaries produce shots, and a Charades-Ego-trained SlowFast classifier produces ego/exo labels per shot. The detector is a multimodal transformer that takes past frames, past narrations, the next narration, and past views as input. The authors show that the pretrained detector can be repurposed as a view selector by injecting candidate ego/exo frames and fine-tuning with limited labels. Evaluation covers view-switch detection on HT100M and zero-shot on Ego-Exo4D, as well as view selection with limited labels on Ego-Exo4D, reporting gains over retrieval, heuristic, and LangView baselines.","tokens_in":21788,"tokens_out":5681,"duration_ms":51646,"significance":"If the evidence held up, the contribution would be valuable: it offers a practical way to learn editing preferences from human-edited videos without manual view labels, with a clear experimental framework, human-annotated test sets with substantial inter-annotator agreement, ablations, and zero-shot transfer evaluation. The strengths are the careful annotation protocol, the breadth of baselines, and the detailed supplementary analyses. However, the central claim depends on the pseudo-labeler's accuracy, which is not validated on the target domain, and the HT100M evaluation protocol may systematically reward agreement with the pseudo-labeler. As a result, the paper currently supports a weaker claim: that a representation trained on pseudo-labeled view-switch events transfers modestly to independent multi-view data.","major_comments":[{"comment":"The training signal for the pretext task is entirely produced by the view pseudo-labeler: PySceneDetect shots are labeled by a Charades-Ego-trained ego/exo classifier. The paper never reports the pseudo-labeler's accuracy on HowTo100M against human labels. Since Charades-Ego is a controlled, paired ego/exo dataset and HowTo100M is in-the-wild with close-up exo shots that can resemble ego views, a systematic bias in the pseudo-labeler would be learned by the detector as if it were human view-preference. Please validate the pseudo-labeler on a human-labeled HT100M subset, reporting precision and recall, ideally stratified by shot type and boundary proximity, and show that training on filtered labels preserves the downstream gains.","section":"Sec. 3.2 / Supp. 7.11.1"},{"comment":"The HT100M test set is constructed by additionally including \"the clips right after all view-switch boundaries, as identified by our pseudo-labeler,\" and the past views for HT100M evaluation are obtained from the same pseudo-labeler. This couples the evaluation to the pseudo-labeler used in training: the model may be rewarded for mimicking the pseudo-labeler's boundary and view decisions rather than for matching human-edited view-switch choices. The large HT100M margins (AUC 63.8 vs. 53.4) could therefore overstate the method's ability to predict human view switches. Please report results on an unbiased HT100M test set, such as randomly sampled clips with human-annotated past views, and treat the Ego-Exo4D zero-shot evaluation with ground-truth past views as the primary quantitative evidence for human view-switch generalization.","section":"Supp. 7.9 / Sec. 5 (Implementation)"},{"comment":"The headline claim of state-of-the-art view selection rests on a 2.5 AUC margin over LangView-bigData in Table 2 and a 3.8 AUC margin in zero-shot view-switch detection in Table 1. These margins are modest, and the ablation \"ours w/o pretraining\" (AUC 51.6) shows that the pretext matters. Given the pseudo-label validation gap above, the current evidence does not yet establish that the model learned human view-switch patterns; it may have learned a generic ego/exo-adapted representation. I would be satisfied if the authors add pseudo-label validation, unbiased HT100M evaluation, and confidence intervals or variance estimates for the zero-shot Ego-Exo4D margin over the best retrieval baseline.","section":"Sec. 5, Tables 1 and 2"}],"minor_comments":[{"comment":"The sentence \"our model inputs do provide cues about the the view type\" contains a duplicated article; please fix.","section":"Sec. 5, View-switch detection paragraph"},{"comment":"The caption contains the typo \"popenopenotentially\"; it should read \"potentially.\"","section":"Fig. 4 caption"},{"comment":"The phrase \"common for both both HT100M and Ego-Exo4D\" repeats \"both\"; please correct.","section":"Supp. 7.10"},{"comment":"These entries refer to the same work with inconsistent author spelling (\"Nagarajan\" vs. \"Nagarjan\"); please unify the citation.","section":"References [34] and [35]"},{"comment":"The choice of a 2-second minimum shot length for PySceneDetect is not motivated; a sentence explaining this choice would help readers assess the pseudo-labeling pipeline.","section":"Supp. 7.11.1"}],"recommendation":"major_revision","confidential_remarks":"I am sympathetic to the authors' overall direction; the problem is well motivated and the experiments are extensive. The main reason for major revision is not a fatal flaw but a missing validation that is central to the training-data claim. The authors' prior LangView work is the strongest baseline, and the paper should more explicitly state what new capability is added beyond that work, namely the ability to learn from unlabeled single-view in-the-wild videos rather than labeled multi-view videos. I would not reject; the requested analyses are feasible within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has a genuinely new pretext task—learn view-switch detection from pseudo-labeled unlabeled web videos, then transfer to view selection with limited labels—and it is delivered with real rigor. The central claim mostly holds, but the HT100M headline numbers are softer than they look because of test construction and unvalidated pseudo-labels. The Ego-Exo4D results are the trustworthy part.\n\nWhat's new: the view-switch detection task itself, and the shot-level pseudo-labeling pipeline that turns unlabeled how-to videos into training signal. That's distinct from LangView's language-based weak supervision, and the comparison against LangView-bigData (which uses 98x more data) is a fair and strong baseline. The paper also does careful human annotation with inter-annotator agreement, multiple ablations, and a zero-shot evaluation on Ego-Exo4D that uses ground-truth past views and human labels. The shot-level pseudo-labeling ablation (Table 4) shows the design choice matters, and the ablations over input modalities are informative.\n\nWhere it gets soft: the stress-test concern is on the money. The pseudo-labeler's ego/exo accuracy on HowTo100M is never validated against human labels. The training task is effectively 'at a detected scene cut, predict whether the next shot's pseudo-labeled view differs.' The model may be learning the pseudo-labeler's biases, including scene-cut detection. Worse, the HT100M test set (Supp 7.9) deliberately oversamples clips right after pseudo-labeler-detected boundaries, and the same pseudo-labeler provides the past views at evaluation time. So the large HT100M margins (AUC 63.8 vs 53.4) are consistent with the model agreeing with the pseudo-labeler, not necessarily with human view preferences. The Ego-Exo4D zero-shot result is cleaner—human labels, ground-truth past views—but the margin is modest (AUC 56.4 vs 53.4), and the fine-tuned view-selection gain over LangView-bigData is about 2.5 AUC. That's a real gain, but it suggests the pretraining helps mainly by adapting representations to ego/exo, not by learning fine-grained human editing patterns.\n\nThere's also no released code or annotations, which makes independent verification harder. The paper does disclose the test construction in the supplementary, which I credit—it's not hidden.\n\nWho it's for: researchers in video understanding, automatic cinematography, and egocentric vision. It's a serious contribution that deserves a serious referee. I'd recommend major revision: ask the authors to validate the pseudo-labeler on HT100M against human labels, and to re-run HT100M evaluation on a test set not biased by pseudo-labeler boundaries. If those numbers hold up, the paper is in good shape. As it stands, the central claim is partially supported—the Ego-Exo4D results carry it, but the strength of the headline is not established.","headline":"Genuinely new pretext task and fair baselines, but the HT100M results are compromised by unvalidated pseudo-labels and test-set construction; the Ego-Exo4D results are cleaner but modest.","tokens_in":22283,"tokens_out":2705,"would_cite":true,"duration_ms":22915,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mining unlabeled web videos teaches a model when to switch camera views.","keywords":["view selection","egocentric video","exocentric view","instructional video","weak supervision","pretext task","view-switch detection","pseudo-labeling"],"falsifier":"Collect a test set where ground-truth view switches are manually annotated across many HowTo100M videos, train the same model with the pseudo-labeler replaced by these clean labels, and check whether zero-shot transfer to Ego-Exo4D collapses or improves; if clean labels do not change or improve performance, the pseudo-labeling pipeline is not the source of the claimed gains.","tokens_in":21272,"feed_emoji":"🎬","tokens_out":3024,"duration_ms":24822,"temperature":0.7,"pith_summary":"The paper tries to show that a view-switch detector trained with pseudo-labels from unlabeled how-to videos can teach a model to pick which camera to show in a multi-camera instructional video, even when few manually labeled examples are available. The authors argue that human-edited web videos encode implicit choices about which viewpoint best shows an activity, and those choices can be recovered without direct supervision. If right, this would make automatic cinematography for how-to videos practical, since the training signal would come from videos that already exist rather than from expensive annotations.","feed_headline":"Unlabeled web clips teach AI which camera to show","feed_subtitle":"How-to videos carry hidden editing choices; a new pretext task mines them to pick the best view.","key_machinery":"The pretext task is two-class next-view prediction: given past frames, past narrations, past views, and the upcoming narration, the model decides whether the next interval is egocentric or exocentric. The training signal comes from a pseudo-labeler that splits each HowTo100M video into shots with PySceneDetect, classifies each shot with a Charades-Ego-trained ego/exo classifier, and averages clip probabilities per shot to label the view sequence. The detector itself is a transformer that fuses DINOv2 frame features, Llama-2 narration features, learned view embeddings, and temporal embeddings through a [CLS] token, and the same architecture with candidate ego/exo frames appended becomes the view selector.","core_discovery":"SWITCH-A-VIEW learns to predict whether the view will change at any moment in an instructional video, then repurposes that detector as a view selector. Trained on 3,416 hours of HowTo100M videos whose ego/exo labels are produced automatically, the detector outperforms retrieval-based and heuristic baselines on view-switch detection, and after fine-tuning on a small labeled set of Ego-Exo4D clips it outperforms the state-of-the-art weakly supervised view selector. The authors report that removing the pretraining step sharply degrades performance, supporting their claim that the pretext task supplies the useful transferable knowledge.","pith_inferences":["If the pseudo-labeler's ego/exo bias carries over, the detector may systematically favor one view type in ambiguous cases; a dataset with clean ego/exo boundaries would test this.","The same pretext idea could be extended to more than two views, e.g., selecting among multiple exo cameras, using continuous view representations instead of a binary label.","The use of the next narration as a cue suggests that offline editing, where the full narration track exists before selection, is the right setting; live streaming would need to drop or predict that input."],"forward_implications":["A model pretrained this way needs as few as 1,000 labeled examples to beat a baseline trained with 98x more data.","View-switch detection transfers zero-shot to Ego-Exo4D, suggesting the learned view patterns generalize across video domains.","Combining the model's pretraining with narration-based pseudo-labels from a prior method yields further gains, indicating the two weak-supervision signals are complementary.","The approach removes the need for negative examples, because unlabeled edited videos supply positive view choices implicitly."],"supporting_citations":[{"why":"Supplies the large-scale unlabeled in-the-wild how-to videos used to train the view-switch detector.","marker":"[37]"},{"why":"Supplies the multi-view Ego-Exo4D videos and best-view annotations used to evaluate zero-shot view-switch detection and few-shot view selection.","marker":"[19]"},{"why":"Supplies the learned ego-exo view classification model that is a component of the pseudo-labeler.","marker":"[29]"},{"why":"Supplies the Charades-Ego dataset used to train the ego/exo classifier in the pseudo-labeler.","marker":"[50]"},{"why":"Provides the state-of-the-art weakly supervised view selector baseline that the paper compares against and whose pseudo-labels it later combines with.","marker":"[34]"},{"why":"Provides the InternVideo2 encoder used in the retrieval and view-narration similarity baselines.","marker":"[59]"}],"fun_headline_variants":["AI learns to pick best camera view from unlabeled how-to videos","Unlabeled videos teach AI to switch camera views automatically","Model learns view-switch timing from raw how-to clips","No labels needed: AI picks the perfect camera shot from web videos","AI watches unlabeled how-tos to decide which camera you see"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pseudo-labeled ego/exo segments in HowTo100M reflect genuine human view-switch preferences, which requires the Charades-Ego-trained classifier and PySceneDetect boundaries to be accurate enough on in-the-wild how-to videos.","fun_headline_variants_meta":{"raw":{"variants":["AI learns to pick best camera view from unlabeled how-to videos","Unlabeled videos teach AI to switch camera views automatically","Model learns view-switch timing from raw how-to clips","No labels needed: AI picks the perfect camera shot from web videos","AI watches unlabeled how-tos to decide which camera you see"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000728,"raw_usage":{"total_tokens":3206,"prompt_tokens":836,"completion_tokens":2370,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":452,"completion_tokens_details":{"reasoning_tokens":2284}},"tokens_in":452,"tokens_out":2370,"duration_ms":12594,"temperature":1.0,"reasoning_tokens":2284,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:41:34.604922+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a test set where ground-truth view switches are manually annotated across many HowTo100M videos, train the same model with the pseudo-labeler replaced by these clean labels, and check whether zero-shot transfer to Ego-Exo4D collapses or improves; if clean labels do not change or improve performance, the pseudo-labeling pipeline is not the source of the claimed gains.","supporting_citations":[{"cited_title":"Howto100m: Learning a text-video embedding by watching hundred mil- lion narrated video clips","cited_arxiv_id":null,"evidence_quote":"Supplies the large-scale unlabeled in-the-wild how-to videos used to train the view-switch detector."},{"cited_title":"Ego-exo: Transferring visual representations from third- person to first-person videos","cited_arxiv_id":null,"evidence_quote":"Supplies the learned ego-exo view classification model that is a component of the pseudo-labeler."},{"cited_title":"Internvideo2: Scaling foundation models for mul- timodal video understanding","cited_arxiv_id":null,"evidence_quote":"Provides the InternVideo2 encoder used in the retrieval and view-narration similarity baselines."}],"review_version":1}