{"id":"e6217223-4c8c-436e-ada4-c7bed6c8be22","arxiv_id":"2412.06182","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A zero-shot method that turns long videos into hierarchical text descriptions and uses them for retrieval and question answering without any fine-tuning.","lead":"This paper builds a pipeline that turns long videos into layered text descriptions using off-the-shelf object detectors, action recognizers, captioners, and an LLM. The text summaries are used directly, with no training, for video retrieval and question answering.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Keyframe-based segmentation may split or merge scenes, so the claimed zero-shot generality of the hierarchical textual representation is not yet established.","rationale":"The paper's own stated goal is a zero-shot, fine-tuning-free video understanding method whose textual representation directly serves retrieval, PRVR, and QA. The strongest claim is therefore only as strong as the clip-level chapters produced by the Bottom-up Video Interpretation Mechanism. The reader's identification of intra-frame segmentation as the weakest assumption is exactly the load-bearing point: Section III-B argues that I-frames are more reliable than motion vectors because they are independently coded, but that property concerns compression fidelity, not semantic coherence. GOP structure is encoder-defined and can split one event across clips or combine two events into one. The paper's Table XI ablation varies sampling strategy, but all variants still operate on the same keyframe-defined clips, so it cannot validate the boundary assumption. The paper also reports no direct measure of segmentation quality for the eight datasets. There are other concerns (no code, no error bars, hyperparameters tuned on test sets, and the odd storage figure), but those are evidence-quality issues; the I-frame boundary assumption is the one that, if wrong, would undermine the central claim itself. It is worth noting the method would not become useless: textual redundancy reduction and the LLM story summarization could partly repair bad boundaries, which is exactly why the concrete test should ask whether the downstream numbers are stable under boundary-choice perturbations. Since the test has not been run, the verdict should remain CONDITIONAL rather than ACCEPT or REJECT.","tokens_in":25852,"tokens_out":1788,"duration_ms":17045,"concrete_test":"Run a controlled comparison on ActivityNet Captions and Charades: for the same videos, generate FDVS representations under (a) the paper's I-frame boundaries, (b) uniform boundaries with the same average clip length, and (c) PySceneDetect semantic boundaries, keeping all other components fixed. If R@1 on PRVR and video QA accuracy change by less than a noise band under repeated LLM runs, the central claim is robust to boundary choice and the concern is retired; if the I-frame variant is materially worse than PySceneDetect or materially different from uniform, the segmentation assumption is load-bearing and the zero-shot claim is not yet supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that FDVS's hierarchical textual representation is directly usable across retrieval, PRVR, and QA with no task-specific fine-tuning. The strongest claim depends on clip chapters being coherent semantic units, and the paper's only mechanism for defining clip boundaries is intra-frame (I-frame) placement (Section III-B). The reader is right that this is the least secure link: I-frame positions are determined by the encoder's GOP structure for compression efficiency, not by semantic boundaries. If a scene change falls between two I-frames, that clip mixes two events; if one long scene contains multiple I-frames, it is split. The paper's defense (Section III-B) is that intra frames are 'independent of inter-frame information' and hence reliable, but independence from motion estimation does not imply alignment with semantic boundaries. The ablation in Table XI compares keyframe-based sampling to uniform sampling and all-frame sampling, but all three use the same keyframe-derived clip boundaries for the goal of isolating clips and the keyframe rows differ by more than just boundary placement, so the ablation does not test whether clip boundaries are semantically meaningful. Because every downstream use (textual redundancy reduction via Sentence-BERT similarities, chapter set C, and story summarization from C) is downstream of this segmentation, a systematic misalignment would change which moments are summarized and which snippets are retrieved for PRVR and QA. The paper reports strong numbers, but no analysis is provided of segmentation quality, e.g., how often a clip contains a scene change, nor any comparison against PySceneDetect or uniform boundaries that are evaluated by downstream performance. The claim of direct applicability without fine-tuning would still be plausible if the representation were robust to segmentation errors, but that robustness is not demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FDVS, a pipeline that converts long untrimmed videos into hierarchical textual representations using only off-the-shelf pretrained modules: keyframe-based clip segmentation, sampled-frame visual redundancy reduction, three perception agents (object detection, action recognition, image captioning), an LLM that writes per-clip \"chapters,\" a textual redundancy reduction stage, and a final LLM-generated \"story.\" These text representations are then used directly, without task-specific fine-tuning, for partially relevant video retrieval, text-to-video retrieval, and video question answering. The method is evaluated on eight datasets across the three tasks, with ablations over sampling strategy, redundancy reduction, perception modules, local memory length, and number of sampled frames. The paper reports strong zero-shot numbers, lower storage costs than deep-feature baselines, and qualitative examples of detailed video understanding.","tokens_in":26096,"tokens_out":6052,"duration_ms":67876,"significance":"If the empirical claims hold, FDVS offers a practical and interpretable zero-shot alternative to learned video features: it introduces no trainable parameters, reuses strong vision and language foundation models, and provides a single hierarchical text artifact that supports retrieval, partially relevant retrieval, and QA. The paper also ships an unusually large set of ablations and reports storage advantages, which are useful for practitioners. However, the central \"no fine-tuning\" and \"zero-shot\" claims are currently weakened by test-set-based hyperparameter selection, single-run stochastic LLM decoding, and an unsupported assumption that encoder I-frames provide semantically coherent clip boundaries. These issues are fixable within the manuscript's scope, but they are load-bearing for the headline generalization claim.","major_comments":[{"comment":"The keyframe-based segmentation is the foundation of the entire hierarchy, but the paper's justification conflates compression independence with semantic coherence. Section III-B argues that intra frames are reliable because they are \"independent of inter-frame information,\" yet I-frame placement is determined by the encoder's GOP structure for random access, not by scene or event boundaries. A scene change can therefore fall inside one clip, and a long scene can be split across multiple clips. Table XI does not isolate this effect: the \"All frames\" and \"Keyframe-based sampling\" rows share the same keyframe-derived boundaries, while \"Uniform sampling\" changes both boundary placement and sampling density. I ask the authors to quantify clip coherence directly (e.g., compare with a scene detector such as PySceneDetect or with annotated boundaries) and to report how retrieval and QA scores change when boundaries are perturbed.","section":"Section III-B and Section IV-F, Table XI"},{"comment":"The headline zero-shot claim is undermined by selecting the two main hyperparameters on the very test sets used to report the headline numbers. Section IV-B states that the local memory length L=35 is chosen \"based on the ablation study in Table XVIII,\" which evaluates video retrieval on ActivityNet Captions, and the default 8 frames per clip is chosen from Table XIX on MSRVTT. These are the same datasets whose final retrieval results appear in Tables VIII and IX. Because the pipeline is otherwise parameter-free, tuning these choices on the test sets can absorb test-specific variance and overstate the zero-shot generalization. Please move hyperparameter selection to a validation split or use a leave-one-dataset-out protocol, and state the selected values explicitly.","section":"Section IV-B, Tables XVIII and XIX"},{"comment":"All quantitative results appear to be single runs even though the LLM is decoded with temperature 0.7. Several headline comparisons are close enough to be within decoding noise (e.g., Table VIII: FDVS 31.6 R@1 vs. HiTeA 29.9; Table VI: FDVS 42.4 vs. Bard+ShortViViT 42.0; Table XVIII: L=31 vs. L=35 differ by 1.1 R@1). Without multiple seeds and variance estimates, or a switch to greedy decoding for the main tables, the reported improvements are not statistically grounded. Please report mean and standard deviation over at least three runs for the central tables, or use temperature 0 and state that the results are deterministic.","section":"Section IV-B and Tables IV-IX"}],"minor_comments":[{"comment":"The caption of Table XV says the ablation is on ActivityNet Captions, but the default rows (CLIP-base, Vicuna-7B: R@1 31.6, R@5 56.8, R@10 65.3) match the MSRVTT numbers in Table VIII rather than the ActivityNet numbers in Table IX (R@1 28.4, R@5 57.5, R@10 71.1). Please correct either the caption or the numbers.","section":"Section IV-F, Table XV"},{"comment":"The prompt templates contain several typos that should be fixed: \"finnally,\" \"winthin,\" \"infomation,\" and \"explaination.\" Minor as they are, they matter because the prompts are part of the method's reproducibility.","section":"Table II"},{"comment":"Table X reports storage in \"bytes\" per video (e.g., 87.3 bytes for FDVS on MSRVTT), but a detailed textual story is far larger than 87 bytes. Please clarify the unit (bytes, kilobytes, tokens, or normalized feature bytes) and state how the number was computed; otherwise the 89.6% storage reduction claim is not verifiable.","section":"Section IV-B and Table X"},{"comment":"Eq. (2) defines a cosine similarity, but the text calls it a cosine distance and then removes frames whose \"similarity distance\" exceeds the threshold. This is confusing: a frame more similar to the keyframe is removed as redundant, which is consistent with the similarity interpretation, but the terminology should be made consistent throughout.","section":"Section III-C, Eq. (2)"},{"comment":"The symbol L is reused for both the number of chapters in Eq. (4) and the local memory length in Section III-E. This creates ambiguity when reading the ablation in Table XVIII. Please use distinct symbols (e.g., K for the number of chapters and L for memory length).","section":"Section III-D, Eq. (4)"},{"comment":"The column \"#V-T Data\" labels FDVS as 0M, which is technically true only for the FDVS pipeline itself. The pipeline uses InternVideo, BLIP2, and GroundingDINO, all of which were pretrained on large video-text or image-text corpora, and retrieval uses AnglE, which is fine-tuned on semantic textual similarity. The 0M label should be qualified so readers do not interpret the comparison as showing that no external data were used anywhere in the system.","section":"Tables IV-VIII"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong central idea and a large amount of empirical work, but the three major comments are interconnected: test-set hyperparameter tuning, stochastic single-run reporting, and the unvalidated I-frame segmentation assumption together make the \"zero-shot, no fine-tuning\" claim stronger than the current evidence supports. These issues are addressable with additional experiments and reporting changes, so I do not recommend rejection, but the revision needs to be substantive rather than purely editorial."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read of arXiv:2412.06182. The core idea—render a long video as hierarchical text (clip chapters plus a whole-video story) and reuse that text for retrieval, PRVR, and QA with no task-specific training—is genuinely useful and not present in LLoVi or VideoAgent. The bottom-up interpretation and the two-stage redundancy reduction are clean, and running the pipeline on eight datasets is a lot of work. I give them that.\n\nBut the evidence as reported is weaker than the headline. L=35 and 8 frames per clip are chosen by ablations on ActivityNet and MSRVTT and then reported on the same datasets. That's test-set tuning. With LLM temperature at 0.7, single-run numbers are unstable; there are no error bars or multiple seeds. The storage claim of 87.3 bytes per video for MSRVTT can't be right—a story is thousands of characters. That looks like a unit or accounting error, and it undermines the storage-efficiency argument until corrected.\n\nThe stress-test concern about I-frame segmentation is real but not fatal. I-frames are encoder places, not scene boundaries, so clips can mix or split events. The paper's ablation compares keyframe-based sampling against uniform sampling, so it does test boundary choices to some degree, but it doesn't measure segmentation quality or robustness to boundary noise. If the representation is robust to mis-segmentation, the claim survives; that robustness isn't shown. A boundary-perturbation experiment or at least a report of how often a clip contains a scene change would address this.\n\nAlso, the comparison with LLoVi and VideoAgent is incomplete—LLoVi appears only on EgoSchema, and VideoAgent is cited but not evaluated. A fair comparison using the same LLM and retrieval protocol would be more convincing.\n\nOverall, the central idea is plausible and deserves a serious referee. It's a conditional accept: fix the storage accounting, add variance estimates, move hyperparameter selection to validation sets, and address segmentation robustness. This is for researchers working on long-video understanding and training-free video-to-text pipelines. I'd send it to peer review, not desk-reject it. I wouldn't cite it in its current form.","headline":"A clever training-free video-to-text pipeline with broad experiments, but the reported numbers are undercut by test-set tuning, no error bars, and an implausible storage figure.","tokens_in":26704,"tokens_out":3635,"would_cite":false,"duration_ms":34711,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a long, untrimmed video can be compressed into hierarchical textual representations—clip chapters plus an overall story—that directly support zero-shot video retrieval, partially relevant retrieval, and question…","keywords":["long video understanding","hierarchical textual representation","zero-shot video understanding","video story generation","redundancy reduction","text-to-video retrieval","video question answering","large language models"],"falsifier":"Take a video with a known scene change that falls between two keyframes—for example, a long single take with no keyframe at the moment the location switches—and check whether FDVS's chapter for that clip describes both scenes; if it does, the segmentation premise fails.","tokens_in":25653,"feed_emoji":"🎬","tokens_out":7305,"duration_ms":67350,"temperature":0.7,"pith_summary":"FDVS is a method for converting a long, untrimmed video into a compact hierarchy of text: a short chapter for each clip and one story for the whole video. The paper's central claim is that this textual representation is rich enough to feed directly into three downstream video tasks—text-to-video retrieval, partially relevant video retrieval, and video question answering—without any task-specific fine-tuning or training on video-text pairs. The method's central move is a bottom-up process of perception, clip-level summarization, and video-level summarization that keeps the details that matter while discarding visual and semantic redundancy. A reader should care because most long-video systems either need labelled data for each task or align vision and language with expensive pre-training, whereas FDVS reuses frozen components and leaves the result as human-readable text. The paper reports results on eight datasets spanning the three tasks, claiming that the zero-shot text representation matches or exceeds several models that were trained on video-text data.","feed_headline":"FDVS turns long videos into text for zero-shot video tasks","feed_subtitle":"A bottom-up pipeline writes clip chapters and a full story, reusing that text for retrieval and QA without retraining.","key_machinery":"The load-bearing object is the Bottom-up Video Interpretation Mechanism, a hierarchical summarization chain in which each level's text is produced by a large language model from the previous level's text, working upward from frame-level perception to clip chapters to the whole-video story. Two redundancy operators support it: visual redundancy reduction removes frames whose image-encoder cosine similarity to the clip's keyframe is high, and textual redundancy reduction removes a chapter whose text-encoder similarity to the running average of recent chapters is high. The remaining chapters are then summarized into a single story, and downstream tasks consume either the story, selected chapters, or both.","core_discovery":"The central claim is that an untrimmed long video can be represented as a hierarchy of text with multi-granularity detail, and that this text alone is sufficient input for several video-understanding tasks. FDVS builds the hierarchy bottom-up: it segments the video at keyframes, prunes visually redundant frames within each clip, extracts object, action, and scene information with frozen perception models, has a language model write a chapter for each clip, prunes textually redundant chapters, and then has the language model summarize the surviving chapters into one long video story. The paper asserts that this hierarchical textual representation carries enough fine-grained and global information that retrieval, partially relevant retrieval, and question answering can be run on it in a zero-shot manner, with no fine-tuning on task-specific datasets.","pith_inferences":["One implicit consequence not tested in the paper is that the same recipe could extend to audio: adding a speech recognizer as another perception agent would let the story text cover dialogue and sound events, a direction the paper itself mentions as a possible optimization.","Because the representation is plain text, the story and chapters could be indexed by standard text search engines, enabling retrieval over very large video collections; the paper demonstrates semantic text retrieval but not this systems-level deployment.","A testable extension is to feed the generated chapters to a temporal localization or summarization task by changing only the prompt template; if the representation is as task-agnostic as claimed, no other change should be needed."],"forward_implications":["A single offline pass over an untrimmed video yields a reusable text artifact that can answer retrieval, partially relevant retrieval, and question answering queries without any per-task training.","Video retrieval reduces to text-to-text retrieval, so any text embedding model can score queries against the story and the chapters.","Long-video question answering becomes a reading task for a language model over the story text, decoupling visual perception from language reasoning.","The paper reports average storage per video of 87.3 bytes on MSRVTT, far smaller than stored deep-feature representations, which would make large video archives cheaper to index.","Because all components are frozen, improving the perception models or the language model should directly improve downstream accuracy, with no extra fine-tuning."],"supporting_citations":[{"why":"Supplies the image encoder whose cosine similarity scores decide which frames are visually redundant.","marker":"[70]"},{"why":"Supplies the per-frame image captions that form the scene-level perception information.","marker":"[65]"},{"why":"Supplies object categories, positions, and sizes for the object-level perception information.","marker":"[71]"},{"why":"Supplies action categories for the temporal-level perception information.","marker":"[54]"},{"why":"Supplies textual embeddings used to measure chapter similarity for semantic redundancy reduction.","marker":"[72]"},{"why":"Supplies the language model that turns perception information into clip chapters and the final video story.","marker":"[87]"}],"fun_headline_variants":["FDVS writes a video's story in text for zero-shot tasks","From clips to a full story: FDVS interprets long videos as text","Video to story: bottom-up text hierarchy for zero-shot QA and retrieval","FDVS: long video becomes a detailed story, no fine-tuning needed","Turn long videos into hierarchical text stories for zero-shot tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the video encoder's keyframes land on real scene or event boundaries, so that each clip between two keyframes contains one coherent scene.","fun_headline_variants_meta":{"raw":{"variants":["FDVS writes a video's story in text for zero-shot tasks","From clips to a full story: FDVS interprets long videos as text","Video to story: bottom-up text hierarchy for zero-shot QA and retrieval","FDVS: long video becomes a detailed story, no fine-tuning needed","Turn long videos into hierarchical text stories for zero-shot tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000339,"raw_usage":{"total_tokens":1830,"prompt_tokens":865,"completion_tokens":965,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":873}},"tokens_in":481,"tokens_out":965,"duration_ms":8208,"temperature":1.0,"reasoning_tokens":873,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:55:00.344573+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a video with a known scene change that falls between two keyframes—for example, a long single take with no keyframe at the moment the location switches—and check whether FDVS's chapter for that clip describes both scenes; if it does, the segmentation premise fails.","supporting_citations":[{"cited_title":"BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the per-frame image captions that form the scene-level perception information."},{"cited_title":"DINO: DETR with improved denoising anchor boxes for end-to-end object detection,","cited_arxiv_id":null,"evidence_quote":"Supplies object categories, positions, and sizes for the object-level perception information."},{"cited_title":"Sentence-bert: Sentence embeddings using siamese bert-networks,","cited_arxiv_id":null,"evidence_quote":"Supplies textual embeddings used to measure chapter similarity for semantic redundancy reduction."}],"review_version":1}