{"id":"5601edaf-331a-4657-ad7b-31ff993aad51","arxiv_id":"2506.07016","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AVHaystacks is a new 3100-question benchmark for audio-visual QA across 500 videos, and the MAGNET multi-agent pipeline beats current baselines on it.","lead":"This paper introduces AVHaystacks, a new benchmark for answering questions that require pulling evidence from many videos using both audio and visuals. It also proposes MAGNET, a multi-agent retrieval pipeline, and reports large gains over earlier baselines on this benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's core premise—QA pairs genuinely require both audio and visual—is asserted but never tested; a transcript-only baseline could collapse the task to text-based reasoning.","rationale":"The most load-bearing concern is the benchmark's validity premise: that AVHaystacksQA questions cannot be answered from transcripts or captions alone. This is exactly the reader's weakest assumption, and the paper provides no experimental control for it. The concern is grounded internally: the benchmark was built almost entirely from transcripts (F.2–F.5), and MAGNET retrieves using caption embeddings as well as audio-visual embeddings (§3.2). The only evidence for modality necessity is an LLM self-report prompt and human filtering, which is not a substitute for a transcript-only or single-modality baseline. If that baseline performs comparably, the task reduces to text-based multi-document QA and the reported gains lose their claimed meaning. I also considered the GPT-judge/meta-agent overlap and the fine-tuning confound in the headline improvements, but those affect the magnitude of the reported gains rather than the validity of the task; the modality-dependence question is more fundamental. Since the reader already issued a CONDITIONAL verdict centered on this same issue, and the recommended condition (adding transcript-only controls and releasing artifacts) is the natural fix, no verdict adjustment is needed.","tokens_in":25524,"tokens_out":4084,"duration_ms":41393,"concrete_test":"Evaluate a transcript-only baseline on AVHaystacks-Full: run GPT-4o or Gemini-1.5-Pro on cleaned, timestamped transcript segments only (no video/audio), with the same QA prompts, and compute the paper's BLEU@4, CIDEr, Text-Sim, GPT-Eval, STEM, and MTGS. Also run audio-only and visual-only variants. Pre-register the criterion: if transcript-only achieves at least 80% of MAGNET+Qwen-2.5-Omni-FT on BLEU@4 and GPT-Eval, or if human raters cannot distinguish transcript-only answers from MAGNET answers at better than chance, then the modality-necessity filter in §2 is not supported and the benchmark/task claims should be re-scoped.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2 states that queries are filtered to 'genuinely require both audio and visual understanding,' but the only filtering evidence given is an LLM prompt (Supplementary Listing 1: 'must be answer by a video, audio and can not be answered by text only?') plus human inspection. In contrast, the construction pipeline is transcript-centric: captions are cleaned and segmented (§2, F.2–F.4), segment-aware questions are generated from transcript segments (Listing 2), and ground-truth answers are step-wise texts with video/timestamp references (Listing 5). The framework itself relies on captions: 'Captions are generated using Gemini 1.5 Pro and encoded with ImageBind' (§3.2). No experiment feeds only transcripts, only audio, or only video to any baseline. If transcript-only input reproduces most of MAGNET's BLEU/GPT-eval/STEM/MTGS scores, then AVHaystacksQA mostly measures text-based retrieval and summarization over captions, not audio-visual reasoning, and the claimed '89%/65% relative improvements' would not support the task as defined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AVHaystacksQA, a multi-video audio-visual question-answering task in which a model must retrieve relevant videos from a large corpus, localize salient temporal segments, and synthesize a grounded step-wise answer. To support this task, the authors present AVHaystacks, a benchmark of roughly 3100 annotated QA pairs built from 500 YouTube videos across 27 topics, and MAGNET, a retrieval-augmented multi-agent framework that combines ImageBind-based audio-visual retrieval, a salient frame selection module (SFS), per-video AVLLM agents (Qwen 2.5 Omni, VideoSALMONN, Unified IO2), and a GPT-4o meta-agent for final answer synthesis. The paper also proposes two evaluation metrics: STEM, a step-wise error metric based on Hungarian matching, and MTGS, a matched temporal grounding score. Experiments compare MAGNET with several baselines on AVHaystacks-50 and AVHaystacks-Full, and ablations study modality contribution, frame sampling, top-k selection, meta-agent choice, and threshold sensitivity.","tokens_in":25774,"tokens_out":6483,"duration_ms":64101,"significance":"If the benchmark's central premise is valid, AVHaystacks fills a genuine gap: no existing benchmark systematically evaluates multi-video audio-visual retrieval and reasoning with temporal grounding, and MAGNET is a practical, model-agnostic pipeline that shows consistent gains across its own ablations, including a human evaluation component. The benchmark construction effort is substantial, and the STEM and MTGS metrics address an under-evaluated aspect of grounded answer generation. However, the significance is conditional: the paper never verifies that the QA pairs actually require both audio and visual information, and the headline gains are reported against zero-shot baselines while MAGNET is fine-tuned, so the claimed improvements cannot currently be attributed to the framework rather than to in-distribution training.","major_comments":[{"comment":"The central premise of AVHaystacks—that each QA pair 'genuinely requires both audio and visual understanding' and cannot be answered from text alone—is asserted but never tested. The only filtering evidence is an LLM prompt (Supp. Listing 1: 'must be answer by a video, audio and can not be answered by text only?') plus human inspection, while the construction pipeline is transcript-centric: captions are cleaned and segmented (§2, F.2–F.4), segment-aware questions are generated from transcript segments (Listing 2), and ground-truth answers are step-wise texts with video/timestamp references (Listing 5). No experiment feeds only transcripts, only audio, or only video to any baseline or to MAGNET. If a transcript-only model reproduces most of the reported BLEU/GPT-eval/STEM/MTGS scores, then AVHaystacksQA mostly measures text-based retrieval and summarization over captions, not audio-visual reasoning, and the claimed 89%/65% relative improvements would not support the task as defined. Please add explicit modality-ablation baselines—transcript-only, audio-only, video-only, and full—and report all metrics for each.","section":"§2, Supp. F.2–F.3"},{"comment":"The central comparison is confounded: fine-tuned MAGNET variants are compared to zero-shot baselines. MAGNET+Qwen-2.5-Omni-FT is LoRA fine-tuned on AVHaystacks (Supp. I), while VideoRAG, Video-RAG, Qwen2.5 Omni, Unified IO2, and VideoSALMONN are evaluated without any fine-tuning. The large relative gains could therefore reflect exposure to the training distribution rather than the multi-agent retrieval architecture. To make the claim that MAGNET 'significantly enhances' performance, the authors should either fine-tune all baselines with the same LoRA recipe or restrict the headline claims to comparisons with equally trained systems.","section":"§4.2, Tables 2–3, Supp. I"},{"comment":"The headline numbers 'up to 89% and 65% relative improvements' are not reproducible from Table 2. On AVHaystacks-50, the best BLEU@4 is 57.67 (MAGNET+Gemini 1.5 Pro) versus 43.16 for VideoRAG, which is a 33.6% relative gain; the best GPT Eval is 8.03 versus 6.32, a 27% relative gain. The corresponding Full-split gains are 34.2% and 27.3%. Please state which baseline and which split yield the 89% and 65% figures, or correct the numbers in the abstract and Section 4.3.","section":"Abstract, §4.3"},{"comment":"The GPT-as-a-judge evaluation is potentially biased because the judge model (GPT) is from the same model family as the meta-agent (GPT-4o, Section 3.3). This creates a same-family write-and-grade loop in which stylistic similarity to GPT-generated text may inflate scores independently of answer correctness. Given that GPT Eval is one of the two headline metrics, the paper should report correlation between GPT Eval and human evaluation, and ideally also include a judge from a different model family (e.g., Gemini or Claude) to verify that the conclusions are not judge-family artifacts.","section":"§4.1, §3.3"},{"comment":"The retrieval evaluation is incomplete and underspecified: Section 4.1 defines R@1, R@3, and R@5, but Table 4 reports only R@3 and R@5, and the row labeled 'Ours' does not specify which MAGNET configuration was used (backbone, top-k, SFS parameters). Please include R@1 and explicitly state the configuration and, if different settings were tried, how the reported point was selected.","section":"§4.1, Table 4"}],"minor_comments":[{"comment":"The text says 'switching from Uniform to SFS increases BLEU@4 score by 0.17', but the corresponding Table 6 values (36.58 to 53.61 for Qwen-2.5-Omni-FT) show an increase of about 17 points, not 0.17. Please correct the decimal error.","section":"§4.4"},{"comment":"The caption refers to 'Tab. 2' for the improvement from the salient frame selection module, but the SFS ablation appears in Table 6. The citation should be updated.","section":"Figure 2"},{"comment":"The abstract and Section 2 state '3100' QA pairs, while the supplementary data (Table 7) sums to 3147 questions. These numbers should be reconciled.","section":"Abstract, §2, Supp. Table 7"},{"comment":"References [8] and [97] are the same work (VideoRAG) and are cited separately; the duplication should be removed. Reference [24] cites a blog URL rather than the official Gemini model documentation or paper.","section":"References"},{"comment":"The related work section uses 'NeXT-QA' while Table 1 and the reference list use 'NExT-QA'; please standardize the spelling.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid engineering contribution, but its two central claims—that AVHaystacks genuinely requires audio-visual reasoning and that MAGNET's headline gains are due to the framework rather than fine-tuning—are not currently supported by the experiments. The 89%/65% headline figures also do not match the reported tables. These issues are fixable within the manuscript's scope, but they require new experiments (modality ablations, controlled baselines) and a corrected presentation of the numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, read this for the benchmark, not for MAGNET. The genuinely new thing is AV-HaystacksQA: 3100 QA pairs over 500 videos, 82% requiring evidence from at least two videos, with audio-visual temporal grounding. That fills a real gap; current benchmarks are mostly single-clip or single-video. The two new metrics, STEM and MTGS, are clearly specified, and the human evaluation (20 raters, Cohen's kappa 0.82) gives some independent support. The ablation coverage is better than average: modalities, sampling, top-k, meta-agent, frame count, threshold, plus an honest failure case. If the benchmark is released, it will be useful.\n\nThe soft spots are real and load-bearing. First, the paper never tests the central premise that each question genuinely requires both audio and visual. The construction pipeline is transcript-centric (segment captions, generate questions from transcript segments), and the filter is an LLM prompt plus human inspection. No baseline is run with transcripts only, audio only, or video only. That is a testable claim, and absent that test, the task may collapse to text-based reasoning over captions. The ablation in Table 5 shows both modalities help inside MAGNET, but that shows the system benefits from modalities, not that the ground-truth QA requires them.\n\nSecond, the headline 89%/65% relative gains are confounded. A fine-tuned MAGNET is compared against zero-shot baselines. Looking at the table, MAGNET with zero-shot backbones is actually worse than VideoRAG on BLEU@4 (29 vs 43). So the gains are largely from fine-tuning, not from the multi-agent framework per se. The paper should either compare fine-tuned baselines or reword the claim.\n\nThird, the GPT judge is from the same family as the meta-agent (GPT-4o). That creates a same-family write-and-grade loop. Human eval mitigates it—the ordering holds—but the headline GPT-eval number should not be read as independent.\n\nFourth, no data or code is released in the paper; reproducibility is limited to the constructed subset AVHaystacks-50, which is better than nothing but insufficient.\n\nThe math is fine: SFS is a straightforward DP, STEM is a sensible Hungarian-matching formulation. Citation pattern is normal, though related work leans heavily on the first author's earlier papers.\n\nWho is this for: anyone building or evaluating multi-video retrieval-augmented QA, and benchmark consumers. It deserves a serious referee, but the referee should demand a transcript-only/single-modality baseline, a fine-tuned-baseline comparison, and artifact release. My recommendation: send to review, with major revision contingent on those experiments.","headline":"This is a benchmark paper worth taking seriously, but the paper's signature claim—that every QA pair genuinely needs both audio and video—is asserted, not demonstrated.","tokens_in":26307,"tokens_out":3201,"would_cite":true,"duration_ms":32104,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces AVHaystacksQA, a task where answering requires retrieving and linking audio-visual evidence across up to 500 videos; it contributes the 3100-pair AVHaystacks benchmark and MAGNET, a multi-agent framework that lifts…","keywords":["audio-visual question answering","multi-video retrieval","temporal grounding","retrieval-augmented generation","multi-agent framework","video benchmark","multimodal LLM evaluation","needle-in-haystack"],"falsifier":"Give a strong text-only LLM the cleaned transcripts (and, optionally, auto-generated captions) of the top-ranked videos for each AVHaystacks question, with no frames and no audio, and score its answers with the paper's own BLEU@4, GPT-judge, and STEM protocol. If transcript-only answers match or exceed MAGNET's scores on a significant sample, the benchmark is answerable from text and the reported audio-visual gains are an artifact of retrieval rather than of multimodal reasoning. A cheaper check is to have annotators judge whether a random sample of 200 ground-truth answers is derivable from the transcript alone.","tokens_in":25345,"feed_emoji":"🎬","tokens_out":14772,"duration_ms":132704,"temperature":0.7,"pith_summary":"The paper argues that real-world video question answering is not a single-clip task: queries about how-to collections, language lessons, and video archives often require pulling evidence from many videos at once, using both sound and picture. To study this, it introduces AVHaystacksQA and a new benchmark, AVHaystacks, made of 3100 question-answer pairs over a pool of 500 videos, where most answers must cite at least two distinct clips. It then proposes MAGNET, a model-agnostic pipeline that retrieves candidate videos, spawns one audio-visual agent per video to locate the salient segments, and lets a meta-agent stitch those segments into a grounded, step-by-step answer. The paper reports that MAGNET improves BLEU@4 by up to 89% and GPT-judge scores by up to 65% over baseline systems, and that two new metrics, STEM and MTGS, expose step-order and temporal-grounding errors that end-to-end scores hide. If the paper is right, video QA should be reframed as a retrieval and multi-hop reasoning problem rather than a single-clip comprehension problem.","feed_headline":"AI agents find audio-visual needles in 500-video haystacks","feed_subtitle":"A 3100-question benchmark forces models to link audio and visual evidence across many clips at once.","key_machinery":"MAGNET is the carrying mechanism: a retrieval-augmented, multi-agent pipeline. Its load-bearing parts are, first, AV-RAG, a two-branch retriever that scores each video by cosine similarity between the query text embedding and both the fused audio-visual embedding from ImageBind and the embedding of a Gemini-generated caption; second, the Salient Frame Selector (SFS), a dynamic-programming routine that picks k representative frames from m sampled candidates by minimizing summed pairwise affinity, where affinity combines visual similarity with a sine-based temporal separation penalty so the chosen frames are both visually diverse and spread across time; and third, the agent layer, in which one Qwen 2.5 Omni instance per shortlisted video emits its candidate segments and partial answers, and a GPT-4o meta-agent aggregates them into the final step-wise answer. The paper also contributes two evaluation objects: STEM, a step-wise error metric that uses Hungarian matching to count missing, hallucinated, and out-of-order steps along with video-ID mismatches, and MTGS, the mean temporal IoU over video IDs that appear in both prediction and ground truth.","core_discovery":"The central claim is that current audio-visual models are poor at multi-video retrieval and reasoning, and that a pipeline built from off-the-shelf components can close much of the gap. Concretely, the authors claim three things: that AVHaystacks is the first benchmark whose answers require linking evidence across multiple videos, with 82% of pairs citing at least two videos from a pool of up to 500; that state-of-the-art LMMs score poorly on it, both at retrieving the right videos and at grounding each answer step in the right time interval; and that MAGNET, a retrieval stage using ImageBind embeddings of fused audio-visual features and of Gemini-generated captions, a salient-frame selector, per-video Qwen 2.5 Omni agents, and a GPT-4o aggregator, raises retrieval recall and answer quality substantially, achieving up to 89% and 65% relative improvements over baselines on BLEU@4 and GPT evaluation scores while nearly matching a Gemini 1.5 Pro upper bound placed inside the same framework.","pith_inferences":["The decisive test the paper leaves unrun is a transcript-only probe: feed cleaned subtitles, with no frames and no audio, to a strong text LLM under the same retrieval ranking and score with the same metrics; if it approaches MAGNET's scores, the benchmark is measuring text reasoning, not audio-visual reasoning.","Because every MAGNET component is model-agnostic, the same retrieve-then-agents-then-meta-agent pattern should transfer to other retrieval-heavy multimodal settings, such as document or image haystacks; the authors list end-to-end trainable modules as future work but do not discuss this transfer.","The ablations on top-k and frame count imply retrieval budget is a tunable policy variable, so plotting answer quality against compute per query would give practitioners an explicit cost-quality curve the paper does not draw."],"forward_implications":["Multi-video audio-visual QA becomes a testable benchmark task, so future LMMs can be measured on retrieval-plus-reasoning rather than single-clip comprehension.","Large gains are available without waiting for end-to-end training: the paper's ablation shows that combining off-the-shelf retrieval, per-video agents, and a meta-agent already lifts answer quality, and fine-tuning the per-video agent pushes an open model close to the closed-source upper bound.","STEM and MTGS give evaluators a way to attribute failures to specific causes: steps that are missing, hallucinated, or out of order, and segments that are mis-grounded in time or in which video they come from.","The modality ablation supports the paper's premise that both channels carry complementary evidence, with audio-plus-visual inputs outperforming either modality alone across every reported metric.","Retrieval behavior matters to answer quality: the top-k and frame-selection ablations show that giving agents more candidate videos and denser frames improves grounding until it plateaus, after which extra input mostly adds compute."],"supporting_citations":[{"why":"Provides the shared ImageBind embedding space that fuses audio and visual features and encodes captions for the retrieval stage.","marker":"[23]"},{"why":"Qwen 2.5 Omni is the audio-visual LLM instantiated as one per-video agent that predicts salient temporal segments and partial answers.","marker":"[25]"},{"why":"GPT-4o acts as the meta-agent that aggregates individual agent outputs into the final grounded step-wise answer.","marker":"[26]"},{"why":"Gemini 1.5 Pro generates the per-video captions whose embeddings drive the caption-retrieval branch of AV-RAG.","marker":"[24]"},{"why":"VideoRAG is the principal long-context video retrieval-augmented baseline whose BLEU@4, retrieval recall, and grounding scores MAGNET is measured against.","marker":"[8]"},{"why":"Video-RAG is the second adapted baseline used to gauge retrieval and QA performance on AVHaystacks.","marker":"[29]"},{"why":"Hungarian matching underpins the STEM metric by pairing predicted and ground-truth steps so missing, hallucinated, and out-of-order steps can be counted.","marker":"[28]"},{"why":"Visual Haystacks supplies the needle-in-a-haystack framing that AVHaystacksQA transfers to multi-video audio-visual retrieval and reasoning.","marker":"[7]"}],"fun_headline_variants":["Multi-agent AI finds needles in 500-video haystacks","Agent framework boosts multi-video QA by up to 89%","New benchmark challenges AI to link audio-visual clues across videos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every AVHaystacks question genuinely needs both audio and visual information and cannot be answered from transcripts or captions alone; the curation pipeline asserts this via LLM prompting and human filtering, and although an ablation shows MAGNET does best with both modalities, the paper never measures how transcript-only or caption-only systems fare on the benchmark.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent AI finds needles in 500-video haystacks","Agent framework boosts multi-video QA by up to 89%","New benchmark challenges AI to link audio-visual clues across videos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000385,"raw_usage":{"total_tokens":2078,"prompt_tokens":1032,"completion_tokens":1046,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":970}},"tokens_in":648,"tokens_out":1046,"duration_ms":11015,"temperature":1.0,"reasoning_tokens":970,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:43:51.607816+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give a strong text-only LLM the cleaned transcripts (and, optionally, auto-generated captions) of the top-ranked videos for each AVHaystacks question, with no frames and no audio, and score its answers with the paper's own BLEU@4, GPT-judge, and STEM protocol. If transcript-only answers match or exceed MAGNET's scores on a significant sample, the benchmark is answerable from text and the reported audio-visual gains are an artifact of retrieval rather than of multimodal reasoning. A cheaper check is to have annotators judge whether a random sample of 200 ground-truth answers is derivable from the transcript alone.","supporting_citations":[],"review_version":1}