{"id":"2663cb59-7927-4f8d-9935-7cfe849499d4","arxiv_id":"2602.22897","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"OmniGAIA couples omni-modal perception with multi-hop reasoning and tool use; OmniAtlas improves Qwen3-Omni from 13.3 to 20.8 Pass@1, but the comparison is confounded by tool access.","lead":"OmniGAIA is a 360-task benchmark that asks AI agents to perceive videos and audio together, reason across several steps, and use web search and code tools to produce open-form answers. The paper also introduces OmniAtlas, a training recipe that lifts an open-source omni-modal model from 13.3% to 20.8% on this benchmark, though part of that gain may come from extra perception tools given only to OmniAtlas.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OmniAtlas improvement may be tool/prompt effect: no matched baseline with read_* tools and OmniAtlas system prompt, so 13.3→20.8 training gain is not isolated.","rationale":"The reader's weakest_assumption was ground-truth validity; I agree that no inter-annotator agreement is reported, but that is an audit gap rather than an identified failure. The more actionable threat to the central claim is the training-evaluation confound, which the reader's rationale also notes. The paper's own appendices document that OmniAtlas has tools and prompts absent from baselines, so Tables 2 and 4 cannot attribute the gain to SFT/DPO alone. The benchmark's hardness (Gemini-3-Pro 62.5 vs. Qwen3-Omni 13.3) is less affected by this confound, which is why I recommend keeping the CONDITIONAL verdict rather than moving to REJECT. The proposed control experiment and train/test-overlap audit would settle the issue. I found no evidence of fraud or intentional obfuscation; this is a standard experimental-isolation problem. The detailed construction pipeline and public code/demo links are real positive evidence, but they do not replace a matched-baseline comparison.","tokens_in":29894,"tokens_out":5307,"duration_ms":53889,"concrete_test":"Run a no-training control: evaluate Qwen3-Omni-30B-A3B in the OmniAtlas harness with the OmniAtlas system prompt (Appendix B.2) and read_image/read_audio/read_video tools exposed (Appendix B.3), but with the base model weights unchanged. If this control matches or approaches 20.8, the recipe's training contribution is not supported; if it stays near 13.3, the training effect is corroborated. Additionally, report the exact overlap between the 2,156 synthesized trajectories (§A.1) and the 360 OmniGAIA evaluation tasks; if overlap exists, re-evaluate on a disjoint held-out set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core recipe claim—OmniAtlas training raises Qwen3-Omni from 13.3 to 20.8 (Tables 2 and 4)—is not causally identified. Section 5.1 says all models get the same external tools (web search, browser, code executor), but OmniAtlas additionally receives active-perception tools read_image/read_audio/read_video (Appendix B.3) and a system prompt instructing the model to use them when media is unclear (Appendix B.2). The untrained Qwen3-Omni baseline is run with the base system prompt and, as reported, without the read_* tooling. The +7.5 absolute gain therefore conflates at least three factors: SFT/DPO training, a larger toolset, and a perception-seeking instruction. The same confound is inherited by Table 4's OmniAtlas-SFT and OmniDPO rows and by the error-analysis claim that OmniAtlas reduces ineffective tool use. If most of the gain comes from additional tools/prompt rather than trajectory SFT + OmniDPO, the central contribution of a training recipe is unverified. This is separate from, and arguably more load-bearing than, the ground-truth audit question (though the missing train/test split is also relevant).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents OmniGAIA, a benchmark of 360 omni-modal agent tasks that require joint perception over video-with-audio or image+audio inputs, multi-hop reasoning, and multi-turn external tool use (web search, browser, code). Tasks are constructed via an omni-modal event-graph pipeline: raw media are mined for structured signals, an event graph is built and expanded with tools, and questions are generated by fuzzifying graph nodes/edges, followed by LLM screening and three-reviewer human verification. The paper also proposes OmniAtlas, a training recipe for open omni-modal models that combines hindsight-guided tree exploration, trajectory-level masked SFT, and OmniDPO for fine-grained error correction. Experiments show that OmniGAIA is difficult: Gemini-3-Pro reaches 62.5 Pass@1 while Qwen3-Omni reaches 13.3, and OmniAtlas improves Qwen3-Omni to 20.8. The paper further provides fine-grained error analysis and a controlled comparison of native vs. tool-based perception.","tokens_in":30205,"tokens_out":7042,"duration_ms":61729,"significance":"If the benchmark's ground truth is sound and the OmniAtlas gain is attributable to the proposed training recipe, this is a valuable contribution: it introduces a harder evaluation axis for omni-modal agents and a practical recipe that improves open-source models. The event-graph construction pipeline is a scalable and interesting methodology, and the paper ships code, a leaderboard, and human-verified tasks. The fine-grained error analysis and the three case studies are useful. However, the central claims currently rest on two unaddressed risks: the benchmark's train/test separation and ground-truth auditability, and the confounding of the OmniAtlas training gain with an expanded tool set and system prompt. These need to be resolved before the benchmark and recipe can be relied upon.","major_comments":[{"comment":"The OmniAtlas gain (+7.5, Qwen-3-Omni 13.3→20.8) is not causally identified. §5.1 says all models receive the same external tools (web search, browser, code executor), but OmniAtlas is additionally given read_image/read_audio/read_video tools (App. B.3) and a system prompt instructing active perception (App. B.2). The base Qwen-3-Omni baseline is run without these tools and with the base prompt. Thus the gain conflates trajectory SFT/OmniDPO with a larger tool set and a perception-seeking instruction. Please provide a matched ablation: train/evaluate OmniAtlas without the read_* tools and with the base system prompt, or give baselines the same tool set/prompt, to isolate the training contribution. The same confound affects Table 4 and the error-analysis claims.","section":"§5.1, Tables 2/4, App. B.2–B.3"},{"comment":"No train/test split is defined. OmniGAIA is described as 360 tasks; Appendix A.1 trains on 2,156 synthesized trajectories and §4.4 refers to a 'training set'. The paper never states that these trajectories are disjoint from the 360 benchmark tasks. If the trajectories are derived from the same OmniGAIA tasks used for evaluation, all OmniAtlas numbers are obtained on training data and the comparison with baselines is invalid. Please specify the split, describe how training tasks are selected, and confirm that no evaluation task appears in the training set.","section":"§3.7, §4.4, App. A.1"},{"comment":"Ground-truth validity rests on three graduate reviewers and LLM screening, but no inter-annotator agreement or independent audit is reported. The screening models (DeepSeek-V3.2, Gemini-3-Pro) overlap with the models used for graph construction (§3.3–3.4), trajectory synthesis (§4.2), and answer judging (§5.1). Since every model score depends on answer correctness/uniqueness, please provide a verification protocol: e.g., a random sample of tasks re-annotated by different annotators, agreement statistics, and an assessment of screening bias (what types of tasks were removed).","section":"§3.6"},{"comment":"The LLM-as-a-Judge (DeepSeek-V3.2) is also a construction/training model, and no human evaluation of judge accuracy is reported. The fallback extracts only the last 20 words of the model output, which may discard a correct answer embedded in a long justification. Please validate the judge against human labels on a subsample and report agreement; also test sensitivity to judge model choice.","section":"§5.1, App. A.2/B.1"}],"minor_comments":[{"comment":"Typo: 'next-hope evidence' should be 'next-hop evidence'.","section":"§3 intro"},{"comment":"Inconsistent model naming: 'Qwen3-Omni' vs. 'Qwen-3-Omni' across the abstract, body, and tables; standardize.","section":"Abstract/body/tables"},{"comment":"The word cloud is difficult to read; a cleaner visualization of query terms would be more informative.","section":"Fig. 3"},{"comment":"Training hyperparameters (learning rate, batch size, warmup, precision, etc.) are omitted; please provide them for reproducibility.","section":"App. A.1"},{"comment":"In the 'Audio and Visual Perception Models as Tools (Input No Media)' row, clarify whether Qwen-3 is the text-only model and how it consumes tool outputs.","section":"Table 3"},{"comment":"The fine-grained error analysis relies on LLM-generated labels; no human agreement is reported, so these numbers should be treated as preliminary or supplemented with an agreement study.","section":"§5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-positioned and likely influential, but the missing train/test split and the tool/prompt confound in the OmniAtlas comparison are load-bearing and need to be resolved before publication. The benchmark ground-truth auditability and judge validation also require attention. The reference list contains a high proportion of self-citations, including several 2026 papers from the same group; some editorial scrutiny of novelty placement is warranted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The benchmark is the real contribution. OmniGAIA fills a gap that no one else has filled: tasks that truly require video+audio or image+audio perception, multi-hop reasoning, and multi-turn tool use, with open-form verifiable answers and a hard enough ceiling that even Gemini-3-Pro lands at 62.5 while open models sit in the teens. The event-graph construction pipeline is a thoughtful way to synthesize these tasks, and the paper is transparent about its process — detailed appendices, tool schemas, prompts, case studies including the OmniAtlas failure. I believe the benchmark itself is solid and useful.\n\nThe training recipe is another story. The headline claim that OmniAtlas lifts Qwen3-Omni from 13.3 to 20.8 is not causally identified. Section 5.1 says all models get the same external tools, but that explicitly covers web search/browser/code executor. OmniAtlas additionally gets read_image/read_audio/read_video and a system prompt that pushes it to call them when media is unclear. The untrained Qwen3-Omni baseline runs without those tools and without that prompt. So +7.5 conflates the SFT/DPO training, the larger toolset, and the prompt. The same confound runs through Table 4 and the error-analysis claims. A reader cannot tell how much of the gain is the tool/prompt itself — which matters, since giving any capable open model a perception-friendly toolset and telling it to use it might produce a comparable jump. Table 3's tool-based perception setting is a different manipulation, so it does not save this.\n\nThe missing train/test split is equally concerning. The paper never states that the 360 benchmark tasks are disjoint from the 2,156 trajectories used for OmniAtlas training. If the training set was constructed from the same source media or even the same QA pairs, some of the improvement could be memorization. This is simple to fix and should be required.\n\nThe ground-truth audit is weaker than claimed. Three graduate reviewers verified each task, but there is no inter-annotator agreement, no quantification of how many tasks were rejected or modified, and the LLM screening uses the same DeepSeek-V3.2 that later judges answers. None of this is fatal — I do not see signs of deliberate contamination — but it is unverified and should be audited in a revision.\n\nBottom line: the benchmark deserves wide use. The OmniAtlas recipe is plausible but currently unproven. I would send this to peer review, and I would insist on a matched baseline — same tools, same system prompt, no training — plus a clear statement of the train/test split and an inter-annotator agreement number. If those come through, this will be a solid dataset paper. As it stands, cite the benchmark, but treat the 13.3→20.8 claim with suspicion.","headline":"A genuinely useful new benchmark, but the OmniAtlas improvement is likely inflated by the read_* tools/prompt confound; the training recipe needs a matched baseline before the numbers can be trusted.","tokens_in":30751,"tokens_out":2193,"would_cite":true,"duration_ms":23750,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OmniGAIA, a 360-task benchmark built from an omni-modal event graph, requires AI assistants to perceive video with audio and image+audio pairs, reason in multiple hops, and use external tools over several turns; on it, Gemini-3-Pro reaches","keywords":["omni-modal agents","benchmark construction","multi-hop reasoning","tool use","event graph","preference optimization (DPO)","perception","video and audio understanding"],"falsifier":"Have five independent annotators attempt a random sample of, say, 50 OmniGAIA tasks using the same tools, blind to the paper's labels; if more than a small fraction cannot reach the labeled answer or produce divergent answers, the uniqueness assumption fails and all reported Pass@1 numbers are suspect.","tokens_in":29782,"feed_emoji":"🤖","tokens_out":6772,"duration_ms":62322,"temperature":0.7,"pith_summary":"OmniGAIA is a benchmark of 360 tasks that for the first time combines omni-modal perception—video-with-audio and image+audio inputs—with multi-hop reasoning and multi-turn use of external tools (web search, browser, code), judged by open-form answers that are verifiable against real-world facts. The tasks are built from a novel omni-modal event graph that links entities and events across modalities, then expands the graph with web evidence and 'fuzzifies' nodes so each question forces the full reasoning path. The paper reports that the benchmark is hard: the best proprietary model reaches 62.5 Pass@1 while the strongest open-source model scores 13.3. It then introduces OmniAtlas, a training recipe—hindsight-guided tree exploration for trajectory synthesis, masked supervised fine-tuning, and OmniDPO for correcting the first error in failed trajectories—which lifts Qwen3-Omni to 20.8 and cuts tool-use and reasoning errors.","feed_headline":"Top AI agent scores 62.5% on new omni-modal benchmark","feed_subtitle":"Open-source models must see, hear, reason, and search the web to pass; a new recipe lifts Qwen3-Omni from 13.3 to 20.8.","key_machinery":"The load-bearing object is the omni-modal event graph: a graph whose nodes are entities and events extracted from video and audio at time-stamped granularity, and whose edges are cross-modal relations. A reasoning agent expands this graph by searching the web, browsing pages, executing code, and retrieving related clips; then 'event fuzzification' hides specific nodes along long paths, forcing any solver to traverse the whole chain. On the training side, the key mechanism is OmniDPO plus masked trajectory SFT: SFT supervises only agent-generated reasoning and tool-call tokens while masking tool observations, and OmniDPO locates the first erroneous step of each failed trajectory and preferenc","core_discovery":"The central claim is that omni-modal agent capability is not a perception problem alone: it requires a closed loop of joint perception, multi-step planning, and tool-based verification, and no previous benchmark measures that combination. The paper establishes this by constructing OmniGAIA so that every task needs cross-modal grounding plus next-hop evidence from web tools to yield a unique open-form answer, and by showing that models trained only for perception collapse on it. It further claims that the OmniAtlas recipe materially improves open models on this axis—Qwen3-Omni from 13.3 to 20.8—and that the dominant failure modes in current agents are ineffective tool use and reasoning errors","pith_inferences":["I would expect the event-graph pipeline to double as a data generator: the same expansion-and-fuzzification construction could synthesize training trajectories for other omni-modal agent tasks, making OmniGAIA both a test and a seed.","The error cascade finding suggests a testable extension: forcing a minimal tool budget, or injecting a 'verify your hypothesis' instruction, may recover much of the hard-task gap without further training.","Because Pass@1 judgments mix exact match with an LLM judge, the reported gaps (13.3 vs 20.8) could be sensitive to judge leniency; an editorial check would be to re-judge all hard-task outputs with a second judge model.","The paper's training claims imply that failure localization matters more than data volume: OmniDPO targets the first error while the rest of the trajectory stays intact, so an ablation that applies DPO to entire failed trajectories instead of prefixes should show smaller gains—this is directly testable."],"forward_implications":["If the benchmark is accepted, omni-modal agents should be evaluated on tool-integrated long-horizon tasks, not just perception, and native perception is the default, since delegating to perception tools costs accuracy and latency (Table 3).","More parameters alone do not fix agentic capability: a 560B model underperforms a 30B model on OmniGAIA, so tool-use policy is a distinct bottleneck.","The OmniAtlas recipe is transferable: on a 7B backbone it raises Pass@1 from 3.6 to 13.3, and on 30B from 13.3 to 20.8, with the largest gains coming from reduced tool misuse.","Hard tasks (multi-hop reasoning over long media) remain open even for the strongest model (38.5% vs 78.7% on easy), so the benchmark has headroom for future agentic RL and perception research."],"fun_headline_variants":["Omni-modal AI agents need tool loop, not just perception","Benchmark: AI agents fail omni-modal tasks without tool reasoning","OmniAtlas lifts Qwen3-Omni from 13.3 to 20.8 on omni-modal test","New omni-modal benchmark exposes agent weakness in tool use","Open-source AI agents get 7.5-point boost with OmniAtlas recipe"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole yardstick presumes the 360 annotated answers are correct, unique, and answerable from the given media plus tools—checked by three human reviewers but without inter-annotator agreement, so an ambiguous or wrong label would shift every model's score and the ordering.","fun_headline_variants_meta":{"raw":{"variants":["Omni-modal AI agents need tool loop, not just perception","Benchmark: AI agents fail omni-modal tasks without tool reasoning","OmniAtlas lifts Qwen3-Omni from 13.3 to 20.8 on omni-modal test","New omni-modal benchmark exposes agent weakness in tool use","Open-source AI agents get 7.5-point boost with OmniAtlas recipe"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1258,"prompt_tokens":746,"completion_tokens":512,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":407}},"tokens_in":490,"tokens_out":512,"duration_ms":4664,"temperature":1.0,"reasoning_tokens":407,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T20:32:42.057356+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have five independent annotators attempt a random sample of, say, 50 OmniGAIA tasks using the same tools, blind to the paper's labels; if more than a small fraction cannot reach the labeled answer or produce divergent answers, the uniqueness assumption fails and all reported Pass@1 numbers are suspect.","supporting_citations":[],"review_version":1}