Omni-Decision: A Progressive Evidence-State Agent System for Omni-Modal QA

Feida Zhu; Junhan Shi; Lingrui Mei; Ming Ma; Steven Hoi; Tianming Yang; Weigao Sun; Yiran Zhong; Yi Zhu

Omni-Decision: A Progressive Evidence-State Agent System for Omni-Modal QA

T0 review · 3 major / 0 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A shared evidence state turns omni-modal QA into query-scoped evidence closure, lifting multi-step accuracy far above free-form agent baselines.

desk verdict Useful systems paper with real gains and honest audits; the state-vs-budget confound is real but does not erase the contribution. read the letter →

arxiv 2607.11433 v1 pith:UAV7PGUF submitted 2026-07-13 cs.AI

Ming Ma , Yi Zhu , Yiran Zhong , Feida Zhu , Weigao Sun , Junhan Shi , Lingrui Mei , Tianming Yang

show 1 more author

Steven Hoi

This is my paper · ORCID

classification cs.AI

keywords omni-modalQAevidence-seekingagentsquery-scopedevidencestateclosuremulti-stepmultimodalreasoningtool-usingOmniGAIAWorldSense

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Omni-modal evidence-seeking QA asks systems to answer questions whose clues are sparsely scattered across video, audio, images, web pages, and computation. Most agent systems leave those clues in free-form histories or tool logs, so they cannot cleanly track what is grounded, what still conflicts, and when the chain is ready to answer. This paper argues that the missing piece is not another role topology or perception model, but an explicit query-scoped evidence state that records confirmed evidence, unresolved conflicts, fact and computation dependencies, and open needs. Omni-Decision is a training-free system in which planning, acquisition, validation, repair, finalization, and insufficient stopping all read the same state digest, while only a deterministic reducer commits normalized observations. On open-world OmniGAIA it reaches 45.6% accuracy (+27.3 points over the official base agent) and transfers to WorldSense at 58.3%; no-state ablations and progress audits support a separable contribution from the shared state interface.

What carries the argument

Query-scoped evidence state St = {Et, Ct, Ft, Ut}: confirmed evidence atoms, unresolved conflicts, external fact and computation slots, and open evidence needs. A bounded state digest conditions the planner, critic, and finalizer; only the reducer commits normalized tool observations and critic verdicts through deterministic field updates, with readiness defined as empty needs, complete dependencies, and no conflicts.

What would settle it

Re-run Omni-Decision and the no-state baseline on the full OmniGAIA split with hard-matched tool budgets, step limits, and identical backends; if the large accuracy gap disappears under equal call counts, the central claim that explicit evidence-state control is the separable driver would not hold.

Watch

Extended reading notes

Core claim

Long-horizon omni-modal evidence seeking is best framed as query-scoped evidence closure rather than longer context or free-form trajectories. Maintaining a structured state of confirmed evidence, conflicts, fact/computation dependencies, and open needs, and conditioning every control decision on that shared view, yields large accuracy gains on multi-step omni-modal QA without training.

Load-bearing premise

The accuracy gains are attributed mainly to the evidence-state interface itself, even though the full system also runs much longer tool trajectories under strong proprietary planner and perception backends, and the no-state ablation is only a fixed subset diagnostic rather than a full matched-budget comparison.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing worth knowing is that this is a clean systems argument with numbers: free-form multimodal agent histories are a bad control substrate, and a query-scoped evidence ledger (confirmed atoms, conflicts, fact/compute slots, open needs) plus a deterministic reducer is a better one. They get 45.6% on OmniGAIA vs 18.3% for the official base agent, transfer to WorldSense at 58.3%, and back that with planner/perception swaps, no-state subset drops, Wilson CIs, human progress audits, and stuck-rate localization.

What is actually new is not “agents with tools” or “critics.” They say so. The contribution is the shared read/write contract: everyone reads the same state_digest; only the reducer commits typed events; readiness and insufficient stop are derived from Ut/Ft/Ct rather than left in dialogue. Appendices ship reducer contracts, need statuses, and full prompts. That is more engineering honesty than most multimodal agent papers.

The soft spot the stress-test flags is real and load-bearing for the causal claim. Omni-Decision averages ~11 tool calls vs ~2.5 for the base agent; max 15 steps with critic-gated repair. The no-state ablation is only on a 120-case subset and is not budget-matched. So “state” and “longer, more careful seeking under strong gpt-5.2 + gemini backends” are partly entangled. I would not oversell the +27 pp as pure ledger magic. I also would not dismiss the ablations: drops appear across three backend configs, audits show many wrong answers still make partial chain progress, and stuck cases often look like recognized unclosed slots the perception backend cannot fill. That is useful diagnosis even if isolation is incomplete. No public code is a practical minus for a systems claim.

Math is light by design (ready/can_advance definitions, field-wise ⊕); citations are appropriate and not circular. This is for people building long-horizon multimodal agents who care about inspectable control and abstention, not for pure perception SOTA hunting.

I would send it to peer review. Ask for a full-split, budget-matched no-state (or free-form history) control and clearer cost reporting; do not desk-reject. Worth reading and, for agent-control work, worth citing.

Referee Report

3 major / 0 minor

Summary. The paper proposes Omni-Decision, a training-free agent system for omni-modal evidence-seeking QA that centers control on an explicit query-scoped evidence state St = {Et, Ct, Ft, Ut} (confirmed evidence, conflicts, fact/computation dependencies, open needs). Planning, validation, repair, finalization, and insufficient stopping all read a shared state_digest; only a deterministic reducer commits normalized tool observations and critic verdicts. On OmniGAIA the system reaches 45.56% accuracy versus 18.33% for the official base agent (+27.23 pp) and 25.51% for OmniAgent; on WorldSense it reaches 58.32% versus 28.06% for OmniAgent. Controlled planner/perception swaps, a 120-case no-state ablation (Table 2), and human progress audits on 40 Medium/Hard cases are offered as evidence that the shared state interface contributes beyond free-form trajectories.

Significance. If the causal claim holds, the work is a useful systems contribution: it reframes long-horizon omni-modal QA as query-scoped evidence closure rather than trajectory or role engineering, and it supplies an inspectable read/write contract that other agent stacks could adopt without new training. Strengths include a clear state formalism (§3.1–3.2, Algorithm 1), deterministic commit semantics, Wilson CIs on the full 360-split (Appendix C), multi-backend diagnostics (Table 2), transfer to a closed-world MCQ setting (WorldSense), and process-level audits that separate partial evidence-chain progress from zero-progress failures (§4.3, Appendices D–E). These are concrete engineering and evaluation assets for multimodal agent research, even if absolute SOTA against native end-to-end models is not claimed.

major comments (3)

[§4.2 Table 2; Appendix F] The central claim that explicit evidence-state control—not merely longer multi-step tool use—drives the OmniGAIA gains is only partially isolated. Appendix F reports 4,038 tool requests (avg. 11.22/query) for Omni-Decision versus 881 (avg. 2.45) for OmniGAIA base, with a max of 15 iterations and critic-gated repair. Table 2’s no-state ablation is restricted to a fixed 120-case subset and does not match tool-call budget or trajectory length. A budget-matched free-form-history (or no-state) control on the full 360-split is needed to seal the separable contribution of St; without it, ready(St)/can_advance and extended seeking remain entangled.
[§4.1–4.2; Table 1–2] Comparisons to Minimal agent (5.56%) and OmniGAIA base (18.33%) understate the risk that gains partly reflect stronger orchestration under gpt-5.2 + gemini-3.1-pro rather than the state object alone. Table 2 shows large backend sensitivity (45.56% → 33.33% → 11.39% when perception/planner degrade), while the state effect shrinks to −3.33 pp under the weakest stack. The manuscript should either (i) report a full-benchmark no-state run with the default backends and matched call caps, or (ii) substantially soften claims that attribute the +27.3 pp primarily to evidence-state control.
[§3.2; Appendix H Table 11–12] Answer readiness ready(St)=(Ut=∅)∧complete(Ft)∧(Ct=∅) is treated as an adequate stopping criterion (§3.2), yet Appendix H documents frequent stuck/forced-answer exits when the critic correctly flags unclosed slots that perception cannot fill. The paper does not quantify how often forced answers under revision/budget limits inflate or deflate official accuracy relative to principled abstention. A breakdown of accepted vs forced vs insufficient outcomes on the full OmniGAIA split would make the control claim falsifiable and clarify whether state-conditioned stopping helps or merely delays unsupported answers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical systems paper with external-benchmark accuracy claims, not a derivation that folds inputs into predictions.

full rationale

Omni-Decision is a training-free agent-system paper. Its load-bearing claims are measured accuracies (45.6% OmniGAIA, 58.3% WorldSense) and ablation/audit support for a shared evidence-state interface. Those quantities are scored against external benchmarks and an official judge (gpt-5.2), not derived from parameters fitted to the same targets. The readiness rule ready(St)=(Ut=∅)∧complete(Ft)∧(Ct=∅) is an operational stopping definition, not a prediction of benchmark accuracy. Trajectory audits use human subgoal annotations with an independent scorer sensitivity check; they diagnose progress, they do not force correctness by construction. Self-citations (e.g., Ma et al. 2026 Dover; Zhu et al. 2025) appear only as related work on agent failure/debugging and do not underwrite uniqueness theorems or forbid alternatives. Skeptical concerns about tool-budget confounding versus the state ledger are experimental-isolation issues, not circular derivation. No self-definitional loop, fitted-input-as-prediction, or load-bearing self-citation chain reduces the central result to its inputs.

Assumptions & free parameters 3 free parameters · 5 assumptions · 2 invented entities

Load-bearing content is mostly engineering and evaluation assumptions, not free physical parameters. The central empirical claim rests on benchmark protocols, proprietary model backends, deterministic reducer semantics after stochastic LLM/tool outputs, and the premise that free-form trajectory baselines fairly isolate the value of St despite unequal tool budgets.

free parameters (3)

max_inference_iterations_T = 15
Main experiments cap the loop at 15 steps; stopping and accuracy depend on this budget choice.
subgoal_weights_in_progress_audit = critical=1.0, non-critical=0.5
Human progress scores weight critical subgoals 1.0 and non-critical 0.5; absolute progress levels are sensitive to this hand choice (Appendix D/E).
temperature_and_backend_selection = temperature=0; planner=gpt-5.2; perception=gemini-3.1-pro
Temperature 0 and fixed gpt-5.2 / gemini-3.1-pro defaults are configuration choices that strongly affect measured accuracy (Table 2).

assumptions (5)

domain assumption Omni-modal evidence-seeking QA is better modeled as query-scoped evidence closure / state maintenance than as context scaling alone.
Stated in §1 and used to motivate the whole architecture; not independently proved, but treated as the design premise.
ad hoc to paper After normalization, reducer field updates are deterministic and are the only writes to St; planner/critic/tools are pure readers or event producers.
§3.2–3.3 and Appendix I define this read/write contract; correctness of inspectability claims depends on it holding in implementation.
ad hoc to paper ready(St)=(Ut=∅)∧complete(Ft)∧(Ct=∅) is an adequate answer-readiness criterion for when to finalize or abstain.
§3.2; many failures are later attributed to unclosed slots under this rule, so the metric of success is partly defined by this predicate.
domain assumption Official OmniGAIA LLM judging and WorldSense option matching are valid proxies for answer quality.
§4.1 evaluation protocol; system comparisons inherit judge and format assumptions.
standard math Standard agent/tool-use and multimodal-understanding background (ReAct-style loops, tool APIs, VLM perception).
Used as prior art rather than re-derived; methods build on cited agent and omni-modal systems.

invented entities (2)

Query-scoped evidence state St={Et,Ct,Ft,Ut} independent evidence
purpose: Serve as the shared control object for planning, verification, repair, finalization, and insufficient stopping.
Core invented interface of the paper; not a physical entity, but a new postulated runtime object whose value is tested empirically.
state_digest / deterministic reducer commit boundary independent evidence
purpose: Bound what modules read and ensure only typed normalized events update evidence fields.
Implementation-level invention that makes trajectories inspectable and ablatable; evidence is experimental rather than external physical measurement.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Omni-Decision: A Progressive Evidence-State Agent System for Omni-Modal QA." pith.science (2026). https://pith.science/paper/UAV7PGUF

@misc{pith2026260711433,
  author       = {Pith},
  title        = {Pith review of: Omni-Decision: A Progressive Evidence-State Agent System for Omni-Modal QA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UAV7PGUF}},
  note         = {Machine review of arXiv:2607.11433}
}

read the original abstract

Omni-modal evidence-seeking QA requires agents to answer questions whose evidence is sparsely distributed across videos, audio, images, web pages, and computation results. Existing agentic multimodal systems often leave evidence in scratchpads, tool trajectories, or free-form histories, making it difficult to track what has been grounded, what remains missing, and when the evidence is sufficient to answer. We propose Omni-Decision, a training-free evidence-state system that turns omni-modal QA into a query-scoped evidence-closure process. For each query, Omni-Decision maintains a structured evidence state containing confirmed evidence, unresolved conflicts, fact and computation dependencies, and open evidence needs. A shared state view conditions planning, evidence acquisition, validation, repair, and finalization. Heterogeneous observations from media, web, computation, and verification modules are normalized, judged, and committed through deterministic state updates. This design enables targeted evidence acquisition, preserves sparse cross-modal cues, and provides inspectable control over repair and stopping. Omni-Decision achieves 45.6% accuracy on OmniGAIA and 58.3% on WorldSense, improving over the baselines by +27.3 and +30.2 percentage points, respectively. No-state ablations and trajectory audits further support the role of explicit evidence-state control in multi-step omni-modal evidence seeking.

Figures

Figures reproduced from arXiv: 2607.11433 by the authors.

**Figure 2.** Overview of the Omni-Decision inference loop. Given the immutable context [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Real-progress audit summary. The figure reports full / partial / zero progress counts for the 40 Medium/Hard cases with human subgoal annotations. We further audit whether final-answer failures correspond to zero progress or partial evidence-chain completion [Zhu et al., 2025, Ma et al., 2026]. On 40 Medium/Hard cases with human subgoal annotations, each reference evidence chain is decomposed into weighted factua… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Per-tool request composition of Omni-Decision on OmniGAIA. The figure characterizes [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references

[1]

has an answer

This audit does not replace the official OmniGAIA judge and does not report a new benchmark accuracy. It answers one mechanistic question: when the final answer is not accepted, has the system still completed part of the real evidence chain? Human review is the primary audit. Independent scoring with the Claude audit model claude-opus-4-6 on the same 40 c...

2026
[2]

These are needs whose upstream dependencies are filled and can be productively pursued now

Check evidence_needs and actionable_need_indices. These are needs whose upstream dependencies are filled and can be productively pursued now
[3]

Prefer filling actionable needs before attempting needs in blocked_need_indices
[4]

Match each actionable need's preferred_source to the most relevant tool
[5]

Check tool_call_counts; if a preferred modality has 0 calls and still has open needs, prioritize it
[6]

Use gap_diagnosis to understand what evidence is still missing or conflicting
[7]

insufficient

When calling web_search_tool, refer to the target need's query_focus and combine it with entities from already-filled anchor needs to form a precise query. Do NOT search the full question text verbatim. Evidence-gathering strategy: - Use video-grounding tools first to collect direct evidence before external search. - When filling an anchor need, use the n...
[8]

You have gathered evidence relevant to the question

Show all 19 references

[9]

If any blocking evidence_need is still unfilled or partial, gather more evidence before finishing unless finish_available is true
[10]

The answer directly addresses the question and does not contradict collected evidence
[11]

verdict":

If the answer checkpoint blocks your answer, revise it to be more specific and evidence-based, not more hedging. PROMPTTEMPLATE: PLANNERUSER Single-video template: Answer this question about a video ({video_length} seconds long). Question: {question} Multi-asset template: Answ...
[12]

Review each evidence need against accumulated evidence (evidence_state_digest, accumulated_observations, latest_observation)
[13]

For each need, determine whether it is now unfilled, partial, filled, or unfillable
[14]

When a need is filled, set filled_by to the tool that provided the evidence
[15]

If a need has been targeted by multiple tool calls without progress, mark it unfillable
[16]

If the current query direction is clearly wrong or too vague, use revised_query_focus to correct it
[17]

If a need was misclassified, use revised_need_role to correct it
[18]

If a downstream need depends on a newly identified prerequisite, use add_depends_on to add that dependency index
[19]

verdict":

Return the full updated list in updated_need_statuses. Need status definitions: - filled: The exact entity, value, or date is locked with sufficient precision to safely use in the final answer or downstream computation. - partial: The direction is correct but precision is insu...

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.

[1] [1]

has an answer

This audit does not replace the official OmniGAIA judge and does not report a new benchmark accuracy. It answers one mechanistic question: when the final answer is not accepted, has the system still completed part of the real evidence chain? Human review is the primary audit. Independent scoring with the Claude audit model claude-opus-4-6 on the same 40 c...

2026

[2] [2]

These are needs whose upstream dependencies are filled and can be productively pursued now

Check evidence_needs and actionable_need_indices. These are needs whose upstream dependencies are filled and can be productively pursued now

[3] [3]

Prefer filling actionable needs before attempting needs in blocked_need_indices

[4] [4]

Match each actionable need's preferred_source to the most relevant tool

[5] [5]

Check tool_call_counts; if a preferred modality has 0 calls and still has open needs, prioritize it

[6] [6]

Use gap_diagnosis to understand what evidence is still missing or conflicting

[7] [7]

insufficient

When calling web_search_tool, refer to the target need's query_focus and combine it with entities from already-filled anchor needs to form a precise query. Do NOT search the full question text verbatim. Evidence-gathering strategy: - Use video-grounding tools first to collect direct evidence before external search. - When filling an anchor need, use the n...

[8] [8]

You have gathered evidence relevant to the question