{"id":"faa8ce0d-c081-459a-9242-ca2750304f6d","arxiv_id":"2509.08621","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AdsQA adds an ad-video question-answering benchmark and ReAd-R, a GRPO-trained model that beats 7B baselines but not larger closed models.","lead":"This paper introduces AdsQA, a video question-answering benchmark built from 1,544 ad videos, with five tasks covering visual concepts, emotions, themes, persuasion tactics, and target audiences. It also presents ReAd-R, a reinforcement-trained video QA model that improves over 7B baselines but trails larger commercial models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ReAd-R's 'state-of-the-art' claim is contradicted by Table 1: it scores 25.0 strict, below Qwen2.5-VL-72B (31.0) and Gemini-2.5-Pro (35.9).","rationale":"The reader's weakest assumption (metadata ground truth) is a valid concern about benchmark validity, and the reader's rationale already notes the SOTA problem. I focus on the table contradiction because it is the most load-bearing: it is an internal inconsistency, not an external assumption, and it directly falsifies the paper's flagship claim. The dataset contribution may survive, and the RL recipe may show a small gain over its base model, but the paper must either drop the SOTA claim or re-run the comparison with matched training/evaluation conditions. This supports the reader's CONDITIONAL verdict rather than a rejection of the whole paper.","tokens_in":26927,"tokens_out":5419,"duration_ms":59919,"concrete_test":"Using the released AdsQA evaluation code, independently evaluate ReAd-R (Qwen2.5-VL-7B) and Gemini-2.5-Pro on the same 600-sample subset with identical frame/voiceover inputs and the same GPT-4o scoring prompt (Table 6). If Gemini's strict or relaxed accuracy is higher than ReAd-R's, the SOTA claim in the abstract is false. Additionally, recompute Table 1's full-set numbers with the provided checkpoints; if Qwen2.5-VL-72B remains above ReAd-R, the claim should be revised to 'competitive among 7B open models' at most.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's headline contribution (Abstract, §1, §6) is that ReAd-R achieves state-of-the-art on AdsQA, 'outperforming strong competitors equipped with long-chain reasoning capabilities... by a clear margin.' This is the central empirical claim. Table 1 directly contradicts it. ReAd-R (Qwen2.5-VL-7B) strict accuracy is 25.0 and relaxed 51.5. Gemini-2.5-Pro achieves 35.9/60.7, Qwen2.5-VL-72B achieves 31.0/55.8, and GPT-4o achieves 29.4/56.6. Even ReAd-R's gain over its own base model (Qwen2.5-VL-7B: 23.0 strict) is only +2.0 points, with no error bars or significance testing. The comparison is also not apples-to-apples: ReAd-R is trained on 500 AdsQA pairs and optimized against the same GPT-4o inclusion/exclusion reward used for final evaluation, while the baselines are evaluated zero-shot. The benchmark itself may be a useful resource, but the SOTA claim is unsupported by the paper's own numbers and should be removed or rewritten.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AdsQA, a video QA benchmark built from 1,544 advertisement videos (10,962 clips, 22.7 hours) with five open-ended question types: visual concept understanding, emotion recognition, theme/message extraction, persuasion strategy mining, and audience modeling. Questions and reference answers are generated by a role-played multi-agent LLM pipeline using uploader-supplied metadata as ground truth, followed by automatic cleaning and manual revision. The paper also proposes ReAd-R, a GRPO-based reinforcement-learning model trained on 500 QA pairs, and benchmarks 14 video/multimodal LLMs. The authors claim AdsQA is the first ad-video QA benchmark and that ReAd-R achieves state-of-the-art performance.","tokens_in":27258,"tokens_out":3756,"duration_ms":42393,"significance":"If the benchmark is valid, AdsQA would fill a genuine gap: most video QA datasets target perceptual content, whereas advertisement understanding requires implicit, associative reasoning about persuasion, emotion, and audience. The dataset is publicly described, the construction pipeline is detailed, and the manual quality-control steps are a positive feature. The ReAd-R training recipe is also potentially reusable. However, the current manuscript's central empirical claim is contradicted by its own table, and the benchmark's evaluation design is self-referential. These issues must be resolved before the contributions can be accepted as stated.","major_comments":[{"comment":"The headline claim that ReAd-R 'achieves the state-of-the-art outperforming strong competitors ... by a clear margin' is directly contradicted by Table 1. ReAd-R (Qwen2.5-VL-7B) reaches 25.0 strict / 51.5 relaxed, while Gemini-2.5-Pro reaches 35.9 / 60.7, Qwen2.5-VL-72B reaches 31.0 / 55.8, and GPT-4o reaches 29.4 / 56.6. The gain over the base model is only +2.0 strict accuracy, with no error bars or significance tests. This claim appears in the abstract, introduction, and conclusion and must be removed or rewritten to reflect the actual ranking.","section":"Abstract, §1, §6, Table 1"},{"comment":"The evaluation is circular. Section 3.2 treats uploader metadata as 'ground-truth information' and uses it to generate QA pairs. The GPT-4o evaluator in Section 5.1 is given the same meta-information alongside the golden answer, and the inclusion/exclusion rules used for final scoring are the same rules used as the reward signal in ReAd-R (Section 4). Thus models that reproduce the metadata-derived answer are rewarded twice: once in training and once in evaluation. This does not measure video understanding independently of metadata. The authors should either evaluate without providing metadata to the scorer, or validate the benchmark against human judgments collected without access to metadata, and report agreement.","section":"§3.2, §4, §5.1, Table 6"},{"comment":"The comparison across models is not apples-to-apples. Gemini-2.5-Pro is evaluated on a 600-sample subset, GPT-4o receives captions from Qwen2.5-VL-72B rather than raw video, and ReAd-R is fine-tuned on AdsQA-derived training data while all baselines are zero-shot. In addition, no confidence intervals or significance tests are reported. The paper should report all models on a common subsample, use the same input modality, and include standard errors or bootstrap intervals. This is necessary before any claim about relative model strength can be assessed.","section":"§5.1, Appendix C.1, Table 1"},{"comment":"The evidence that ReAd-R improves over its base model is weak and brittle. The strict-accuracy gain is +1.4 for Qwen2-VL and +2.0 for Qwen2.5-VL, with no variance estimates. The ablation shows that the method fails with uncurated data and with a strict reward, and is sensitive to prompt templates. These results suggest the method's success depends on narrow conditions. The claim that RL is superior to SFT for this task is based on a single configuration; a more systematic comparison with multiple seeds and data subsets is needed.","section":"§5.2, §5.3, Table 2"}],"minor_comments":[{"comment":"The 'first advertisement video QA benchmark' claim should be qualified, since the Pitt dataset [24] already contains a video advertisement subset, as the paper itself notes in §2. The novelty should be framed as the first large-scale, publicly available, multi-task ad-video QA benchmark.","section":"§1, §2"},{"comment":"Typo: 'contibute' should be 'contribute'.","section":"§6"},{"comment":"The relaxed-accuracy parameter λ=0.5 is fixed with no sensitivity analysis. Please report how results change for other values, or justify the choice.","section":"Eq. (4), §5.1"},{"comment":"Table 6 caption says 'Relexed Acc' — should be 'Relaxed Acc'.","section":"Table 6 caption"},{"comment":"The human performance numbers in Table 1 are based on 40 videos / 200 QA pairs, and Gemini-2.5-Pro on 600 samples; the table should clearly mark subset sizes so readers do not interpret all rows as full-dataset results.","section":"Appendix C.1, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely salvageable as a benchmark resource, but the current framing as a state-of-the-art method should be abandoned or substantially revised. I would also ask the authors to clarify the relationship of AdsQA to the ICCV 2025 MARS2 workshop competition, since using the same test set for a public challenge and for a paper's evaluation may create unintended leakage or conflicts. The dataset provenance and licensing terms for the crawled videos also deserve careful verification before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: AdsQA is a real contribution, and you should know about it. The SOTA claim for ReAd-R is not supported by the paper's own numbers. That needs to be fixed before anyone takes the method results seriously.\n\nWhat is genuinely new: this is the first ad-video QA benchmark for LLMs, built from 1,544 ads (22.7 hours) with five open-ended tasks that target implicit reasoning—emotion, persuasion, audience modeling—not just visual perception. The multi-agent annotation pipeline is thoughtful, the human checking is real (five annotators, two rounds, 37% retention), and the task design is more interesting than typical surface-level video QA. GPT-4o at 29.4% strict accuracy shows the benchmark is hard in a useful way. That part of the paper deserves credit.\n\nThe soft spots are significant, and they cluster around the evaluation. First, the abstract and conclusion claim ReAd-R achieves state-of-the-art \"by a clear margin,\" but Table 1 shows ReAd-R (Qwen2.5-VL-7B) at 25.0 strict, well below Gemini-2.5-Pro at 35.9 and Qwen2.5-VL-72B at 31.0. Even against its own base model the gain is only +2.0 points strict, with no error bars or significance testing. That is not a SOTA claim; at best it is a modest improvement over the same-scale base model.\n\nSecond, the evaluation has a circular design. Golden answers are generated from uploader metadata, and the GPT-4o scorer is provided with the same metadata as \"ground truth.\" The reward signal for ReAd-R uses the same inclusion/exclusion rules and metadata. So training the model against AdsQA is partly training it to reproduce metadata, not to reason about the video. Baselines are evaluated zero-shot, while ReAd-R trains on 500 pairs from the same distribution, so the comparison is not apples-to-apples.\n\nThird, uploader metadata from adsoftheworld.com is treated as ground truth in Section 3.2. That metadata is promotional, occasionally incomplete, and written by the ad's creator. Human review mitigates but does not eliminate the risk that answers and scores inherit those biases.\n\nWho is this for? The video QA community and anyone working on ad understanding. The benchmark itself is useful enough to warrant a serious referee, but the paper needs major revision: remove or re-scope the SOTA claim, evaluate with blinded human scoring or metadata-free answers, add error bars, and either train baselines on the same data or clearly label the comparison as zero-shot vs. fine-tuned. As it stands, I would not cite the ReAd-R numbers, but I would cite AdsQA as a dataset.","headline":"AdsQA is a genuinely useful new benchmark, but the headline SOTA claim is contradicted by the paper's own Table 1, and the evaluation loop around uploader metadata undercuts the numeric comparisons.","tokens_in":27752,"tokens_out":2168,"would_cite":true,"duration_ms":25073,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces AdsQA, a video question-answering benchmark built from advertisements, and argues that ad videos expose a distinct kind of implicit, associative reasoning that current video LLMs largely fail at; it also proposes ReAd-","keywords":["advertisement video understanding","video question answering benchmark","implicit multimodal reasoning","reinforcement learning reasoning","persuasion strategy mining","audience modeling","open-ended QA evaluation","multi-agent annotation"],"falsifier":"Take a random subset of AdsQA videos, have human annotators write answers from the video alone with no metadata, and compare those answers to the dataset's golden answers. If human-video-only agreement with the metadata-based keys is low, the benchmark is measuring metadata recall rather than video understanding. Separately, replace the LLM-based evaluator with a different judge and check whether ReAd-R's reported margin over the reasoning baselines survives.","tokens_in":26868,"feed_emoji":"🎬","tokens_out":7977,"duration_ms":76539,"temperature":0.7,"pith_summary":"The paper argues that advertisement videos are a distinct test bed for large multimodal models because understanding an ad requires implicit, associative reasoning—reading symbolism, persuasion tactics, and intended audiences—rather than the explicit step-by-step logic used in math and code. To make that case, it builds AdsQA, a benchmark of 1,544 ad videos (22.7 hours, 10,962 clips) with thousands of open-ended QA pairs across five tasks: visual concepts, emotion, theme, persuasion strategy, and audience modeling. It then proposes ReAd-R, a reinforcement-learned reasoner that produces a think-then-answer response and is trained with a rule-based reward and GRPO on only 500 QA pairs. The paper reports that current top models score around 29–35 percent strict accuracy on AdsQA, while ReAd-R, starting from a 7B backbone, beats the chain-of-thought and search baselines it compares against. The paper itself notes that most QA pairs were LLM-generated and that model-based evaluation can diverge from human judgment.","feed_headline":"Video AI gets ads' hidden meaning right only 29% of the time","feed_subtitle":"A new benchmark asks models to read symbolism, persuasion, and audience; a 7B reward-trained model beats reasoning baselines.","key_machinery":"The load-bearing object is AdsQA's five-task open-ended QA format, built from a modality-interleaved video representation: each ad is split into scene clips, each clip is paired with keyframes, a model-generated description, and transcribed speech, all wrapped with the uploader's metadata. A role-played multi-agent annotation pipeline recruits simulated marketing, design, and psychology experts to write and revise questions, followed by automated reward-model filtering and manual review. The method side rests on a reward-driven reasoning loop: a policy model writes free-form think and answer tags, a rule-guided LLM evaluator scores the answer with inclusion and exclusion rules (1 for full ma","core_discovery":"The central claim is that advertising videos make implicit multimodal reasoning measurable: a model that can describe the objects in an ad can still fail to say what those objects symbolize or who the ad is for. AdsQA formalizes this with five open-ended tasks, and the paper reports that strong commercial models score in the 29–35 percent strict range, well below human performance on a sampled subset. The method claim is that this reasoning can be improved without step-by-step supervision: ReAd-R is trained with outcome rewards and format rewards on 500 QA pairs, raising its 7B backbone's relaxed accuracy from 41.0 to 51.5 when paired with a newer backbone, while supervised fine-tuning on th","pith_inferences":["A testable extension is to isolate the role of metadata: run the same AdsQA questions with uploader metadata removed from the evaluator prompt and measure how much model scores and human agreement drop; this would bound how much of the benchmark's difficulty comes from the video alone versus the supplied description.","The role-played multi-agent annotation pipeline could be reused for other implicit-reasoning domains, such as political ads, satire, or art interpretation, where surface content and intended meaning diverge.","The 0.5 partial-credit reward is a soft target; a stricter reward that penalizes hallucinated brand claims or audience names could reveal whether ReAd-R's gain comes from genuine reasoning or from format compliance.","The paper's human subset is only 40 videos and 200 QA pairs; a larger human-rating campaign could convert AdsQA's model-score gaps into human-meaningful thresholds for what counts as understanding an ad."],"forward_implications":["If AdsQA is a valid probe, then object-level video benchmarks overstate multimodal models' comprehension, and ad-style implicit-reasoning tasks should be part of video understanding evaluations.","If ReAd-R's gain replicates, small models can acquire implicit reasoning from a few hundred outcome-reward examples, without chain-of-thought supervision or process rewards.","The reported negative result for supervised fine-tuning implies that on open, abstract QA, imitation on limited high-quality data can overfit, while reinforcement learning generalizes across diverse ads.","AdsQA's five-task taxonomy gives advertising and media analysis a shared benchmark for creativity, persuasion, emotion, and audience targeting, not just content recognition.","Because AdsQA is released with download scripts rather than video files, future models can be independently scored on it, provided the test set is kept out of their pretraining data."],"supporting_citations":[{"why":"Supplies the source of all ad videos and the uploader-written metadata treated as ground truth for QA generation and evaluation.","marker":"[1]"},{"why":"Provides the frontier multimodal model used for translation, annotation support, a strong baseline, and the free-text answer evaluator.","marker":"[23]"},{"why":"Establishes the earlier image-and-video ad-understanding dataset whose limitations AdsQA claims to overcome.","marker":"[24]"},{"why":"Provides the reinforcement-learning recipe that motivates reward-driven reasoning without process supervision, which ReAd-R adapts.","marker":"[21]"},{"why":"Serves as the base 7B video-language model for ReAd-R and as a primary baseline in the experiments.","marker":"[60]"},{"why":"Supplies a stronger 7B and 72B video-language model used as a baseline, a caption generator, and a backbone for the best ReAd-R variant.","marker":"[4]"},{"why":"Provides the reward model used to automatically filter low-quality QA pairs during benchmark cleaning.","marker":"[80]"},{"why":"Inspires the role-played multi-agent annotation framework and serves as a reasoning baseline in the experiments.","marker":"[78]"},{"why":"Supplies the chain-of-thought video reasoning baseline that AdsQA's results are compared against.","marker":"[15]"},{"why":"Provides the Monte Carlo tree search reasoning baseline that ReAd-R is compared against.","marker":"[84]"}],"fun_headline_variants":["Ads reveal AI's hidden-reasoning gap","LLMs fail ads' symbolism test, new model wins","Video AI can't read ads' hidden logic — yet","AdsQA: Testing AI on persuasion, not just pixels","Reward-trained 7B cracks ad video reasoning"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the uploader-written metadata for each ad is accurate enough to serve as ground truth for both the golden answers and the model-based scoring; if that metadata is incomplete, promotional, or inaccurate, AdsQA's difficulty and model rankings inherit its errors.","fun_headline_variants_meta":{"raw":{"variants":["Ads reveal AI's hidden-reasoning gap","LLMs fail ads' symbolism test, new model wins","Video AI can't read ads' hidden logic — yet","AdsQA: Testing AI on persuasion, not just pixels","Reward-trained 7B cracks ad video reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000144,"raw_usage":{"total_tokens":1038,"prompt_tokens":794,"completion_tokens":244,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":176}},"tokens_in":538,"tokens_out":244,"duration_ms":3280,"temperature":1.0,"reasoning_tokens":176,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T20:15:38.850659+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random subset of AdsQA videos, have human annotators write answers from the video alone with no metadata, and compare those answers to the dataset's golden answers. If human-video-only agreement with the metadata-based keys is low, the benchmark is measuring metadata recall rather than video understanding. Separately, replace the LLM-based evaluator with a different judge and check whether ReAd-R's reported margin over the reasoning baselines survives.","supporting_citations":[],"review_version":1}