{"id":"44cadffe-85a3-4c43-8d2d-487cdd4323e6","arxiv_id":"2411.18180","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"DistinctAD improves automatic movie audio description quality and distinctiveness by adapting CLIP to movie data and using expectation-maximization attention plus a distinctive-word loss over consecutive clips.","lead":"This paper introduces DistinctAD, a two-stage system that generates audio descriptions for movies by adapting a CLIP vision encoder to movie data and then using a context-aware module to avoid repeating descriptions of nearby clips. It reports consistent benchmark improvements, especially on a metric that measures whether each description matches its own clip rather than its neighbors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Distinctive-word loss Eq. (12) is not a valid training objective as printed; it sums log-probabilities of every distinctive word at every token position, conflicting with the autoregressive loss that the paper's R@k/N gains depend on.","rationale":"Both the reader and I identify the same load-bearing concern. The paper's headline contribution is distinctive AD generation, and the two Stage-II mechanisms are the Contextual EMA and the distinctive-word loss. The EMA component is described in enough detail to be reproduced, but the Ldist equation is not. The printed form is not merely ambiguous: under a standard softmax LM, summing -log P(w_n = w_i^d) over all distinctive words at each output position is an invalid objective because the events are mutually exclusive; it would encourage a flat distribution over the distinctive set and degrade fluency. The magnitude (N*u terms) also dwarfs Lauto. Since Table 5 attributes a substantial fraction of the R@k/N improvement to Ldist, the central claim that explicit distinctive-word supervision works is not properly supported until the objective is corrected or code is provided. I would not reject the paper on this basis: the empirical results may well be real, and the issue may be a notation or implementation omission. But the published method cannot be reproduced as written, so the appropriate verdict remains CONDITIONAL, meaning the reader's verdict is unchanged. The concrete test I propose would settle whether the concern lands by checking whether the literal loss behaves as the paper claims.","tokens_in":21476,"tokens_out":7445,"duration_ms":69778,"concrete_test":"Re-run the Stage-II ablations in Table 5 with Ldist implemented exactly as Eq. (12) (e.g., on a single MAD subset with GPT-2), and compare R@5/16 and token-entropy against Lauto-only training. If the literal implementation fails to reproduce the reported C3 > B3 gain or causes degenerate flat predictions, the printed loss is not the real objective, and the authors should provide the corrected equation or release the training code.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (12) defines Ldist = -\\Sigma_{n=1}^N \\Sigma_{i=1}^u log P(w_n = w_i^d | prompt, w<n). At each token position n, this adds a term for every distinctive word in w_d, so the model is asked to maximize the probabilities of u different tokens simultaneously. With a softmax head these probabilities compete, and the objective is minimized by flattening the distribution over the distinctive vocabulary rather than by generating the actual next token. This directly conflicts with the autoregressive loss Lauto in Eq. (11), and because u can be large the Ldist term can dominate Lauto. No masking, indicator, reweighting, or implementation detail is given to make the equation consistent with next-token generation. The paper's central distinctiveness claim rests on this loss: in Table 5, adding Ldist improves R@5/16 by about 3-5 points (e.g., C3 vs B3: 51.7 vs 48.0; C2 vs B2: 51.5 vs 46.1). If Eq. (12) is not the actual objective, the mechanism behind those gains is not described in the paper, and the method cannot be reproduced from the text. This is a correctness gap in the method description, not just a missing baseline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents DistinctAD, a two-stage framework for automatic audio description (AD) generation for movies. Stage-I adapts a CLIP vision encoder to movie-AD data via a global video-AD contrastive loss and a fine-grained frame-AD multi-instance matching loss, claiming to close the domain gap without requiring extra AD text corpora. Stage-II processes N consecutive video clips with a Contextual Expectation-Maximization Attention (EMA) module and an explicit distinctive word prediction loss, using a frozen LLM (GPT-2 or LLaMA) to generate ADs. The authors report state-of-the-art results on MAD-Eval, CMD-AD, and TV-AD, with particular gains on the Recall@k/N metric.","tokens_in":21786,"tokens_out":6637,"duration_ms":60420,"significance":"If the proposed training scheme is sound, the paper makes a meaningful contribution to AD generation: the Stage-I adaptation is a lightweight, reusable way to reduce the movie-AD domain gap, and the Contextual EMA offers a principled mechanism to compact redundant visual information across clips. The empirical study is extensive, covering three benchmarks, multiple backbones, and component ablations, and the reported gains over strong baselines such as AutoAD-II/III are substantial. The paper also includes qualitative comparisons and a motivating analysis of the CLIP embedding space. However, the central distinctiveness mechanism is not reproducible from the published equations, and the paper does not address the significant overlap between the proposed loss and the headline evaluation metric.","major_comments":[{"comment":"The distinctive-word loss as printed is not a valid autoregressive training objective. At each token position n, the double sum over i=1..u of log P(w_n = w_i^d | prompt, w_<n) asks the model to maximize the probability of all u distinctive words simultaneously, but a softmax output layer normalizes over the vocabulary, so these probabilities compete. This conflicts with the standard autoregressive loss Lauto in Eq. (11), and since u can be large, Ldist can dominate the total loss, potentially destroying next-token generation. Moreover, no masking, position alignment, indicator function, or reweighting is specified to connect the set w_d to the actual target tokens, and the term \"word\" is not mapped to the LLM's subword vocabulary. The paper must provide a corrected equation and exact implementation details (how w_d is filtered, how words are tokenized, and how the loss is applied in training). Without this, the gains attributed to Ldist in Table 5 (e.g., C3 vs B3: 25.5 vs 23.3 CIDEr and 51.7 vs 48.0 R@5/16) cannot be traced to the stated mechanism.","section":"Section 3.2, Eq. (12)"},{"comment":"The distinctive-word loss and the R@k/N metric are closely aligned by design: Ldist explicitly trains the model to produce words that are unique to the current AD among its N neighboring context ADs, while R@k/N evaluates retrieval of the current ground-truth AD among those same N neighboring GT texts using token-overlap-based BERTScore. This overlap means that gains on R@k/N are partly a direct consequence of optimizing for the same notion of distinctiveness, and the paper does not discuss this alignment. To make the distinctiveness claim more convincing, the authors should report an independent evaluation (e.g., human ratings of distinctiveness, or a diversity/novelty metric computed on the generated outputs) and should analyze how much of the R@k/N improvement is due to the explicit loss versus the contextual EMA module, ideally with a version of the model trained without Ldist evaluated on a distinctiveness metric not aligned with the training signal.","section":"Sections 3.2 and 4.1"}],"minor_comments":[{"comment":"The abstract states that DistinctAD \"consistently outperforms baselines,\" but in Table 2 on CMD-AD, AutoAD-III† reports a higher CIDEr (25.0 vs 22.7). The authors acknowledge this in Section 4.2, but the wording in the abstract should be qualified, e.g., \"outperforms previous methods that do not use large-scale external pre-training.\"","section":"Abstract and Table 2"},{"comment":"The definition of R@k/N is incomplete: the paper says it \"calculates the average value of Recall@k for each AD with its N temporally adjacent GT texts\" using BERTScore, but does not specify the retrieval procedure (e.g., how candidates are formed, whether the current GT is included, and how ties are broken). Please provide an exact formula or a more detailed description, since this is a headline metric.","section":"Section 4.1, Metrics"},{"comment":"The motivating analysis states that ADs reconstructed from CLIP visual features are \"often of poor quality,\" but only the text-reconstruction results are shown in Table A.1. To support the claim that the vision encoder is the bottleneck, the authors should include the corresponding visual-reconstruction experiment.","section":"Appendix A"},{"comment":"The fine-grained frame-AD matching loss samples a single frame f from Vi; please clarify how many frames are sampled per clip and whether this is done repeatedly per batch, as this affects the gradient estimate and the choice of negative set.","section":"Section 3.1, Eq. (3)"},{"comment":"The hyperparameter settings for the Contextual EMA are not fully explained: the temperatures τ in Eqs. (2) and (6) are not given numerical values, and the x-axis labels in Fig. 4 are not shown for all panels. Please specify the used values in the implementation details and ensure the plots are self-contained.","section":"Section 3.2 and Fig. 4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important problem, and the empirical results are promising. However, the Eq. (12) issue is severe: as written, the loss is not a valid training objective, and the paper provides no implementation details to resolve it. This must be fixed before the distinctiveness claim can be evaluated. The metric-alignment concern is also worth flagging to the authors, as it affects the interpretation of the headline R@k/N numbers. I recommend major revision rather than rejection because the central idea is defensible and the issue is fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what's actually new: DistinctAD is, as far as I can tell, the first system to explicitly target distinctiveness across dense consecutive movie clips for audio description generation. The two-stage design is clean: Stage-I adapts the CLIP vision encoder to the movie-AD domain without needing extra AD corpora, using both global video-sentence contrastive matching and fine-grained frame-word multi-instance matching. Stage-II applies EMA to contextual clips to strip redundant visual features, plus a distinctive-word prediction loss. That last component is the problem.\n\nThe paper does a lot of things well. The Stage-I adaptation is well motivated by a nice diagnostic in the appendix: CLIP text embeddings of ADs can be reconstructed by a simple projector plus GPT-2, while CLIP visual features cannot, which localizes the misalignment to the vision encoder. The authors also did real legwork pulling raw frames for MAD and checking temporal alignments against IMDB; that's the kind of engineering that usually goes unreported. The evaluation is broad: three benchmarks, multiple baselines, and ablations that show consistent gains from both stages. The qualitative examples are convincing — the generated ADs name specific objects (\"phone,\" \"pill bottle\") instead of repeating \"looks at something.\"\n\nNow the soft spots. The biggest one is Eq. (12). As printed, Ldist sums log-probabilities of every distinctive word at every token position. That is not a valid training objective: the probabilities compete under the softmax, and the loss would conflict with the autoregressive Lauto. The stress-test note is right. My guess is the authors meant something like an indicator function over positions where the distinctive words actually occur, or a multi-label classification applied at the appropriate time steps — but they don't say. Since Table 5 attributes roughly 3–5 R@5/16 points to this loss, the mechanism behind those gains is unexplained until the equation is corrected or the implementation is detailed.\n\nTwo smaller issues: hyperparameters (α, β, K, N, γ) appear to be selected on the evaluation set, and no error bars are reported, so it's hard to know how much of the gap is tuning luck. Also, the R@k/N metric and the distinctive-word loss are aligned by design — that's the point, but it means the metric doesn't independently validate the distinctiveness mechanism.\n\nOverall, this is a legitimate contribution to AD generation and long-form video understanding. The central idea holds up; Eq. (12) is a fixable but load-bearing error. I'd send it to peer review, but the reviewers should insist on a corrected loss formula, validation-based hyperparameter selection, and ideally code or variance estimates.","headline":"Solid empirical paper on distinctive AD generation, but Eq. (12) is misprinted or wrong and must be fixed before the method is reproducible.","tokens_in":22306,"tokens_out":3843,"would_cite":false,"duration_ms":34310,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DistinctAD: Generating movie audio descriptions that avoid repeating the same words across similar consecutive scenes.","keywords":["audio description","movie narration","contextual redundancy","distinctive captioning","Expectation-Maximization Attention","CLIP adaptation","Recall@k/N","vision-language models"],"falsifier":"Run the Stage-II training with the printed $L_{\\text{dist}}$ term replaced by a loss that, for each token position, sums over all non-repeated distinctive words as in Eq. 12, and observe whether the model's generated ADs become incoherent or the Recall@5/16 gain over the baseline disappears; conversely, if the paper were to release the exact training code, comparing the loss computation in the released code against Eq. 12 would settle whether the reported gain is attributable to $L_{\\text{dist}}$ as described.","tokens_in":21257,"feed_emoji":"🎬","tokens_out":3798,"duration_ms":28236,"temperature":0.7,"pith_summary":"Audio description (AD) for movies and TV narrates what happens visually between lines of dialogue. Generating ADs automatically is hard because (i) pretrained vision-language models were trained on web images, not movies, and (ii) consecutive clips in a movie often look alike, so a model that sees only one clip at a time tends to repeat the same description over and over. This paper proposes a two-stage system, DistinctAD, that first adapts a CLIP vision encoder to the movie-AD domain without needing extra AD text corpora, then adds a module that strips common visual bases from a block of consecutive clips and an extra loss that rewards predicting clip-specific words. The paper claims that on MAD-Eval, CMD-AD, and TV-AD benchmarks, DistinctAD beats prior automatic AD systems, with the largest gains on Recall@k/N, a retrieval-style metric that directly measures whether a generated AD picks out the right clip among temporally neighboring ones. If correct, the method is a step toward ADs that are not just competent captions but actually distinctive, which matters for blind and low-vision audiences who rely on AD to follow what is happening.","feed_headline":"Movie audio descriptions finally say what is new in each scene","feed_subtitle":"A two-stage system beats prior automatic narrators by stripping redundant context and rewarding clip-specific words.","key_machinery":"The central objects are (a) a CLIP-AD adaptation stage, which fine-tunes only the CLIP vision encoder using two contrastive losses (global video-to-AD matching and frame-to-word multi-instance loss) while keeping the CLIP text encoder frozen, and (b) a Contextual Expectation-Maximization Attention (EMA) module operating on N consecutive video clips, which iteratively estimates responsibilities, updates K compact bases, reconstructs denoised features $\\hat{H}$, and additionally attends from raw features to bases to produce $\\hat{e}_H$ features. A distinctive word loss $L_{\\text{dist}}$ over a word set created by filtering out duplicates from the N-context ADs is added to the standard auto-regressive loss.","core_discovery":"The central claim is that the two failure modes of automatic audio description, a vision-side domain gap and contextual redundancy between neighboring clips, can be addressed by (1) fine-tuning only the CLIP vision encoder with a global video-AD contrastive loss plus a parameter-free fine-grained frame-word multi-instance loss, and (2) adding a Contextual Expectation-Maximization Attention module that compresses and re-attends N consecutive clips' features through K learned bases, together with a distinctive word prediction loss that explicitly pushes the frozen LLM to generate words unique to the current AD rather than words repeated in nearby ADs. The paper reports that this pipeline, with the adapted CLIPAD-B16 encoder and LLaMA3-8B, reaches CIDEr 27.3 and Recall@5/16 56.0 on MAD-Eval and beats AutoAD-III's CIDEr and Recall@1/5 on CMD-AD and TV-AD. The qualitative examples show that prior systems tend to output generic actions like 'looks at something' while DistinctAD tends to name specific objects and actions that identify the current clip.","pith_inferences":["The paper's own experiments include a non-consecutive-clips control (Table 6) showing that R@5/16 *improves* when the N clips are not consecutive; the paper attributes this to more diverse visual content, which would suggest distinctiveness and local accuracy are at least partially in tension.","The distinctive-word loss, as printed in Eq. 12, sums log-probabilities of every distinctive word at every token position, which as written would fight the auto-regressive loss at each position, so the published equation probably does not mirror the actual training objective and would need clarification before the result can be reproduced.","Because the paper downloads raw movies and re-aligns timestamps, the reported results implicitly depend on its own re-sorted MAD split (482 movies, 326,632 ADs rather than the original 488 movies and 334,296 ADs), so direct comparisons with numbers reported by other papers on MAD-Eval may not all be on the same training data.","The method still leaves a large gap to human annotations (CIDEr ~27 vs human-like reconstruction CIDEr 612 from the paper's own Appendix A), so the practical outcome is better *relative* AD quality within a still-far-from-human regime."],"forward_implications":["If the central claim holds, automatic AD can move from generic clip-level captions to context-aware narration that flags what changes from one scene to the next, improving accessibility for blind and low-vision viewers and also supporting eye-free watching.","The adapted CLIP vision encoder (CLIPAD) is a drop-in replacement that can replace CLIP in other CLIP-based partial-fine-tuning AD systems, so the Stage-I adaptation should transfer to other captioning or narration pipelines.","The Contextual EMA and distinctive-word loss are generic enough that the same redundancy-removal recipe could be applied to dense video captioning and to any long-video-to-text task where adjacent segments are semantically redundant.","The distinctive-word loss could be made even more explicit by decoding with a constraint that actively avoids the words already present in the neighboring ADs.","If the evaluation metric Recall@k/N is trusted as a proxy for distinctiveness, the gains reported here (e.g., 56.0 vs 52.8 for AutoAD-III on MAD-Eval) suggest that the method's main contribution is real, measurable improvement in clip-specificity."],"supporting_citations":[{"why":"AutoAD provides the MAD-Eval benchmark protocol and the contextual-AD prompting baseline that DistinctAD extends.","marker":"[20]"},{"why":"AutoAD-II introduces the Recall@k/N metric and the character-aware prompting setup that DistinctAD adopts and improves.","marker":"[21]"},{"why":"AutoAD-III supplies the CMD-AD benchmark, the LLM-AD-eval metric, and the strongest partial-fine-tuning baseline that DistinctAD compares against.","marker":"[22]"},{"why":"AutoAD-Zero provides the character prompts and face-token inference results used directly in DistinctAD's pipeline, plus the TV-AD benchmark.","marker":"[81]"},{"why":"CLIP is the vision-language backbone that Stage-I adapts and Stage-II uses as the frozen/adapted feature extractor.","marker":"[57]"},{"why":"MAD-v2-Named supplies the training ADs; the paper re-downloads raw frames from these movie IDs for Stage-I fine-tuning.","marker":"[67]"},{"why":"The Perceiver adapter from Flamingo is the learnable resampler that maps frame embeddings to T' prompt vectors in Stage-II.","marker":"[4]"},{"why":"Expectation-Maximization Attention (EMA) is the base algorithm that the Contextual EMA module adapts to consecutive movie clips.","marker":"[34]"},{"why":"GPT-2 is used both as the small LLM in the AD-reconstruction experiment (Appendix A) and as one of the decoders in the main experiments.","marker":"[56]"}],"fun_headline_variants":["AI describes movie scenes by what's new, not what's repeated","DistinctAD: Audio descriptions that focus on distinctive details","Teaching AI to describe movie scenes with fresh context","Scene-specific audio descriptions: new method beats prior models","DistinctAD trims redundant context for better movie narration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the distinctive-word prediction loss, as implemented, truly improves distinctiveness without corrupting normal sentence generation, but the paper only prints a loss formula that appears to conflict with the auto-regressive loss and gives no implementation detail to resolve the discrepancy.","fun_headline_variants_meta":{"raw":{"variants":["AI describes movie scenes by what's new, not what's repeated","DistinctAD: Audio descriptions that focus on distinctive details","Teaching AI to describe movie scenes with fresh context","Scene-specific audio descriptions: new method beats prior models","DistinctAD trims redundant context for better movie narration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00066,"raw_usage":{"total_tokens":3054,"prompt_tokens":1016,"completion_tokens":2038,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":1958}},"tokens_in":632,"tokens_out":2038,"duration_ms":12800,"temperature":1.0,"reasoning_tokens":1958,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:26:21.351026+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Stage-II training with the printed $L_{\\text{dist}}$ term replaced by a loss that, for each token position, sums over all non-repeated distinctive words as in Eq. 12, and observe whether the model's generated ADs become incoherent or the Recall@5/16 gain over the baseline disappears; conversely, if the paper were to release the exact training code, comparing the loss computation in the released code against Eq. 12 would settle whether the reported gain is attributable to $L_{\\text{dist}}$ as described.","supporting_citations":[{"cited_title":"Autoad: Movie description in context","cited_arxiv_id":null,"evidence_quote":"AutoAD provides the MAD-Eval benchmark protocol and the contextual-AD prompting baseline that DistinctAD extends."},{"cited_title":"Autoad ii: The sequel-who, when, and what in movie audio description","cited_arxiv_id":null,"evidence_quote":"AutoAD-II introduces the Recall@k/N metric and the character-aware prompting setup that DistinctAD adopts and improves."},{"cited_title":"Autoad iii: The prequel-back to the pixels","cited_arxiv_id":null,"evidence_quote":"AutoAD-III supplies the CMD-AD benchmark, the LLM-AD-eval metric, and the strongest partial-fine-tuning baseline that DistinctAD compares against."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"CLIP is the vision-language backbone that Stage-I adapts and Stage-II uses as the frozen/adapted feature extractor."},{"cited_title":"Mad: A scalable dataset for language grounding in videos from movie audio descriptions","cited_arxiv_id":null,"evidence_quote":"MAD-v2-Named supplies the training ADs; the paper re-downloads raw frames from these movie IDs for Stage-I fine-tuning."},{"cited_title":"Expectation-maximization attention net- works for semantic segmentation","cited_arxiv_id":null,"evidence_quote":"Expectation-Maximization Attention (EMA) is the base algorithm that the Contextual EMA module adapts to consecutive movie clips."},{"cited_title":"Language models are unsu- pervised multitask learners","cited_arxiv_id":null,"evidence_quote":"GPT-2 is used both as the small LLM in the AD-reconstruction experiment (Appendix A) and as one of the decoders in the main experiments."}],"review_version":1}