PlanRAG-Audio: Planning and Retrieval Augmented Generation for Long-form Audio Understanding

Chien-yu Huang; Grant P; Grant P Strimel; Jing; Jing Liu; Liu; Markus M\"uller; Masao Someki; Nathan; Nathan Susanj

arxiv: 2605.20414 · v2 · pith:GTMPJVJLnew · submitted 2026-05-19 · 📡 eess.AS

PlanRAG-Audio: Planning and Retrieval Augmented Generation for Long-form Audio Understanding

Masao Someki , Chien-yu Huang , Siddhant Arora , Samuele Cornell , Markus M\"uller , Nathan Susanj , Rupak V Swaminathan , Grant P Strimel

show 12 more authors

Jing Liu Shinji Watanabe Nathan Susanj Rupak V Swaminathan Grant P Strimel Jing Liu Shinji Watanabe

This is my paper

Pith reviewed 2026-05-21 06:51 UTC · model grok-4.3

classification 📡 eess.AS

keywords long-form audioretrieval-augmented generationplanningaudio reasoningtemporal spansmodality selectionlarge audio models

0 comments

The pith

PlanRAG-Audio plans required modalities and time spans then retrieves only relevant segments to improve accuracy on long audio.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that large audio language models can reason more reliably over long recordings if they first decide which types of information and which time periods matter for a query, then pull only those pieces from a prepared database instead of ingesting the full audio. This matters because current models lose accuracy or become too costly once audio grows long and the needed clues such as speech content, speaker identity, emotion, or sound events are scattered across time. If the claim holds, models would maintain steady performance on complex queries while the amount of data sent to them stays bounded by the query rather than the recording length. A reader interested in practical audio analysis would see this as a way to handle meetings, broadcasts, or surveillance audio without the current length barriers.

Core claim

PlanRAG-Audio is a planning-based retrieval-augmented generation framework that explicitly plans which modalities and temporal spans are required for a given query and retrieves only query-relevant information from a structured text and audio database rather than processing entire recordings directly. This retrieval planning enables effective reasoning over complex, cross-domain audio queries while substantially reducing the input length passed to the large language models. Experiments show improved reasoning accuracy and stabilized performance as audio duration increases by decoupling inference cost from raw audio length.

What carries the argument

The explicit planning of modalities and temporal spans followed by targeted retrieval from a structured text and audio database.

Load-bearing premise

The structured database must contain every query-relevant acoustic cue and the planning step must correctly select the needed modalities and time spans without leaving out critical details.

What would settle it

A set of queries where the planner omits a time span containing a decisive sound event or emotion and the model then gives wrong answers that a full-audio baseline would have answered correctly.

Figures

Figures reproduced from arXiv: 2605.20414 by Chien-yu Huang, Grant P, Grant P Strimel, Jing, Jing Liu, Liu, Markus M\"uller, Masao Someki, Nathan, Nathan Susanj, Rupak V, Rupak V Swaminathan, Samuele Cornell, Shinji, Shinji Watanabe, Siddhant Arora, Strimel, Susanj, Swaminathan, Watanabe.

**Figure 2.** Figure 2: Audio database construction. Raw audio is processed by task-specific modules to construct a structured, time-aligned audio database D(a) consisting of modality-specific metadata streams [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Figure 3: Relative performance of base tasks under long-form audio inputs. Results are for QA-1, MCQA, Summ, [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

read the original abstract

Long-form audio understanding poses significant challenges for large audio language models (LALMs) due to the extreme length of audio sequences and the need to reason over heterogeneous acoustic cues distributed over time, such as speech content, speaker identity, emotion, and sound events. To address these challenges, we propose \textbf{PlanRAG-Audio}, a planning-based retrieval-augmented generation framework for scalable long-form audio understanding. Rather than having audio LALMs process entire recordings directly, PlanRAG-Audio explicitly plans which modalities and temporal spans are required for a given query, and retrieves only query-relevant information from a structured text and audio database. This retrieval planning enables effective reasoning over complex, cross-domain audio queries while substantially reducing the input length passed to the large language models. Experiments across a wide range of speech/audio retrieval demonstrate that PlanRAG-Audio improves reasoning accuracy and stabilizes performance as audio duration increases by decoupling inference cost from raw audio length.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

PlanRAG-Audio adds a planning layer to RAG for long audio but the gains hinge on unverified planner recall of distributed cues.

read the letter

The main takeaway is that this paper puts forward PlanRAG-Audio, a framework that plans which audio modalities and time windows to retrieve from a structured database before passing anything to the language model. The goal is to keep compute from growing with raw audio length while still handling queries that mix speech content, speaker identity, emotion, and sound events spread across long recordings. That framing directly targets a real bottleneck in current large audio language models. The approach is new in its explicit separation of planning from retrieval and reasoning, and it is presented as a procedural method rather than another end-to-end fine-tune. If the planning step works reliably, the reported stabilization of accuracy with increasing duration would be a practical win for applications like meeting analysis. The paper earns credit for naming the heterogeneous-cue problem clearly and for showing that retrieval can be scoped to modalities and spans instead of uniform chunking. The experiments are described as covering a range of retrieval tasks and showing accuracy gains, which at least gives a starting point for comparison. The soft spot is exactly the one the stress test flags: everything rests on the planner correctly surfacing all query-relevant segments. The abstract gives no implementation details for the planner, no recall metrics against ground-truth relevant spans, and no ablation that measures what happens when a critical cue is missed. Without those checks, it is difficult to know whether the stability comes from better context selection or from other factors in the setup. Minor gaps include the lack of baseline descriptions in the summary, but those can be filled in review. This paper is for researchers working on scalable audio understanding and retrieval-augmented methods. A reader who needs concrete ideas for reducing input length on long recordings will find usable pieces even if the planner evaluation needs tightening. It is coherent enough on its own terms to merit a serious referee who can ask for planner diagnostics and fuller experimental controls.

Referee Report

2 major / 2 minor

Summary. The paper proposes PlanRAG-Audio, a planning-based retrieval-augmented generation framework for long-form audio understanding. Instead of feeding entire recordings to large audio language models (LALMs), the method explicitly plans which modalities and temporal spans are needed for a query, retrieves only the relevant information from a structured text and audio database, and performs reasoning on the reduced context. The central claim is that this improves reasoning accuracy and stabilizes performance as audio duration grows by decoupling inference cost from raw audio length.

Significance. If the planning step reliably recovers all query-relevant acoustic cues (speech content, speaker identity, emotion, sound events) distributed over time, the framework would provide a practical route to scalable long-form audio reasoning without quadratic growth in context length. The approach is procedurally novel in its explicit modality-and-span planning layer, but its impact hinges on empirical verification of that layer.

major comments (2)

[Abstract] Abstract: the claim that experiments 'demonstrate accuracy gains and stabilized performance' is unsupported by any reported baselines, datasets, metrics, controls, or statistical tests. Without these details it is impossible to determine whether the observed improvements are attributable to the planning-retrieval mechanism or to other factors.
[Method / Experiments] Method and Experiments sections: the headline accuracy and stability improvements rest on the planner correctly identifying all query-relevant modalities and temporal spans. The manuscript provides no implementation details for the planning module, no error analysis of planner omissions, and no ablation that measures recall of ground-truth relevant segments. If even one critical cue (e.g., a brief overlapping sound event) is missed, the subsequent retrieval supplies incomplete context that the LALM reasoning step cannot recover.

minor comments (2)

[Method] Clarify how the structured database is constructed and indexed, including the exact representation of acoustic cues and the retrieval mechanism (e.g., embedding model, similarity metric).
[Experiments] Figure captions and experimental tables should explicitly state the audio durations tested and the number of queries per duration bin to allow readers to assess the 'stabilization with duration' claim.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed review of our manuscript on PlanRAG-Audio. We have carefully considered the major comments and provide point-by-point responses below. Where the comments identify areas needing clarification or additional analysis, we outline specific revisions that will be incorporated into the next version of the paper.

read point-by-point responses

Referee: [Abstract] Abstract: the claim that experiments 'demonstrate accuracy gains and stabilized performance' is unsupported by any reported baselines, datasets, metrics, controls, or statistical tests. Without these details it is impossible to determine whether the observed improvements are attributable to the planning-retrieval mechanism or to other factors.

Authors: We agree that the abstract, due to its length constraints, does not explicitly list the experimental details. In the revised manuscript we will update the abstract to briefly reference the evaluation datasets, the primary metrics (accuracy and stability measures), the baseline methods compared against, and note that improvements are statistically significant. The full experimental protocol, controls, and results remain in the Experiments section; the abstract revision will better anchor the high-level claim to the reported evidence. revision: yes
Referee: [Method / Experiments] Method and Experiments sections: the headline accuracy and stability improvements rest on the planner correctly identifying all query-relevant modalities and temporal spans. The manuscript provides no implementation details for the planning module, no error analysis of planner omissions, and no ablation that measures recall of ground-truth relevant segments. If even one critical cue (e.g., a brief overlapping sound event) is missed, the subsequent retrieval supplies incomplete context that the LALM reasoning step cannot recover.

Authors: This is a fair and important observation. The effectiveness of PlanRAG-Audio indeed depends on the planner's ability to surface all necessary cues. In the revised version we will expand the Method section with concrete implementation details of the planning module (including the prompting strategy and decision criteria for modality and span selection). We will also add a dedicated error analysis of planner omissions together with an ablation that reports recall of ground-truth relevant segments. These additions will directly quantify the planner's reliability and address the risk of missing critical acoustic cues. revision: yes

Circularity Check

0 steps flagged

No circularity: procedural framework with experimental claims

full rationale

The paper introduces PlanRAG-Audio as an explicit planning-plus-retrieval procedure that selects modalities and temporal spans before querying a structured database, then feeds the results to an LALM. No equations, fitted parameters, or self-referential definitions appear in the provided text. Performance improvements are asserted via experiments rather than derived from prior fits or self-citations. The central claim therefore remains independent of its own inputs and does not reduce by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The framework rests on the existence of a well-structured database and on the planning module's ability to select relevant information; no free parameters or invented physical entities are mentioned.

axioms (1)

domain assumption A structured text and audio database exists that stores query-relevant information extractable without loss of critical cues.
The retrieval step depends on this database being available and complete for the queries tested.

invented entities (1)

PlanRAG-Audio planning module no independent evidence
purpose: To decide which modalities and temporal spans are required for a given query before retrieval.
New component introduced to orchestrate retrieval for long audio.

pith-pipeline@v0.9.0 · 5745 in / 1333 out tokens · 35421 ms · 2026-05-21T06:51:13.369971+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

PlanRAG-Audio explicitly plans which modalities and temporal spans are required for a given query, and retrieves only query-relevant information from a structured text and audio database.
IndisputableMonolith/Foundation/DimensionForcing.lean alexander_duality_circle_linking unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

the system first plans which modalities (e.g., spoken content, speaker information, emotional cues, and non-verbal acoustic events), temporal spans, and constraints are required

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.