{"id":"b45759ca-fafd-452b-83ef-88f67c7289fe","arxiv_id":"2507.00669","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Audio-3DVG grounds spoken descriptions to 3D objects using object-class detection, an object-mention auxiliary task, and audio-conditioned attention, beating the prior audio-only baseline.","lead":"The paper introduces Audio-3DVG, a model that locates objects in 3D point clouds from spoken descriptions instead of typed text. It reports state-of-the-art audio-based grounding results and creates synthetic and real spoken-language benchmarks for 3D scenes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All primary results use Spark-TTS synthetic audio; the only real-world test (Table 7) lacks baselines and is unreleased, so the unvalidated synthetic-to-real transfer is the load-bearing assumption behind the SOTA claim.","rationale":"The Reader's weakest_assumption is exactly the condition that must hold for the central performance claim to carry real-world meaning: Spark-TTS audio is a faithful proxy for actual spoken queries. All training data and the main benchmark tables use synthetic audio, so the reported state-of-the-art numbers are conditional on that proxy. The only counter-evidence check, Table 7, is too thin to validate the transfer: 549 samples spread over three datasets yields wide confidence intervals; there are no baselines on those recordings; and the data are not publicly released, precluding independent verification. If synthetic and real speech differ along dimensions wav2vec 2.0 is sensitive to—background noise, accent variation, speaking rate, disfluencies—then the headline advantage over AP-Refer and the competitiveness with text-based methods may not persist in deployment. This is not a claim of scientific misconduct or a fatal flaw; the method could transfer perfectly well. But the paper currently provides no evidence that it does. The reader's conditional verdict is the right one, and the concrete test above would either strengthen it to acceptance-ready or reveal that the practical claim is overstated. Secondary issues noted by the reader, such as unreported loss weights and the missing OMD ablation, affect reproducibility and internal attribution but are less load-bearing than the synthetic-to-real gap, which threatens the external validity of the central claim itself.","tokens_in":31296,"tokens_out":3228,"duration_ms":42753,"concrete_test":"Release the 549 real-world recordings and run three systems on the identical real-audio split under the same evaluation protocol: Audio-3DVG (trained on synthetic audio), AP-Refer, and a text baseline such as InstanceRefer fed Whisper transcriptions. If Audio-3DVG's real-audio accuracy remains close to its 40.02 synthetic number and its margin over AP-Refer is similar to Table 1, the proxy concern is resolved. If the margin collapses or the Whisper-based text pipeline overtakes it, the synthetic-to-real assumption fails. Also report Whisper WER on real versus synthetic audio to quantify the acoustic gap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's claim that Audio-3DVG 'achieves new state-of-the-art performance in audio-based grounding' is established entirely on synthetic audio: Tables 1, 2, 5, and 6 are trained and evaluated on Spark-TTS renderings of ScanRefer/Nr3D/Sr3D text (§4). The only real-speech evidence is Table 7, with 549 samples, no baseline methods run on those recordings, and the dataset marked for release only 'upon acceptance'. If Spark-TTS audio is acoustically easier than real queries—cleaner prosody, no disfluencies, consistent voices, minimal noise—then the reported 40.02 acc@0.25 and the 16.9-point margin over AP-Refer measure performance on TTS voices, not on the voice-driven interaction the paper motivates. The paper's own limitation section acknowledges dependence on TTS quality, but it does not supply a same-protocol real-audio comparison to quantify the gap. Thus the central claim holds only if synthetic speech is a faithful proxy for real speech; that condition is the weakest link and is currently supported only by an unreleased, baseline-free pilot test.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Audio-3DVG proposes a two-stage audio-based 3D visual grounding framework. It uses PointGroup for instance segmentation, wav2vec 2.0 with a BiGRU for audio encoding, an audio classification head, a multi-label Object Mention Detection head, and an Audio-Guided Attention module with self- and cross-attention conditioned on audio features. The model is trained with three losses: audio classification, Object Mention Detection, and object grounding. Audio data is synthesized from the text of ScanRefer, Nr3D, and Sr3D using Spark-TTS, and a small (549-sample) real-speech test set is collected. On ScanRefer, the model reports 40.02 acc@0.25 overall, surpassing the prior audio-only method AP-Refer (23.09) and the text-based InstanceRefer (37.69); on Nr3D and Sr3D it reports 37.4 and 48.3 overall accuracy, respectively, which are comparable to or better than text baselines. Ablations show the Audio-Guided Attention module and Spark-TTS audio are beneficial. The paper also provides a response-time comparison across text, ASR-transcription, and end-to-end audio pipelines.","tokens_in":31499,"tokens_out":6755,"duration_ms":78263,"significance":"If the reported results hold, the paper demonstrates that direct audio-to-3D grounding without ASR can surpass prior audio-only methods and approach text-based performance, which would be a meaningful step for practical voice-driven interaction. The paper uses standard splits and metrics, includes ablations (Tables 5 and 6), and provides a real-speech test set, albeit a small one. The main contributions—Object Mention Detection and Audio-Guided Attention—are well motivated and clearly described. The significance is, however, conditional on the synthetic-to-real transfer being validated, which is currently not convincingly shown: all primary results are on TTS voices, and the only real-speech evaluation lacks baselines and is not released.","major_comments":[{"comment":"All training and the primary evaluation are conducted on Spark-TTS synthetic audio, while the only real-speech evaluation (Table 7) contains 549 samples, reports no comparison with AP-Refer or text-based baselines on the same recordings, and is explicitly not released. The abstract's state-of-the-art claim for audio-based grounding is therefore established only for TTS voices. This is load-bearing for the paper's practical claim of voice-driven grounding. Please provide a same-protocol comparison on the real recordings (e.g., running AP-Refer and at least one text-based method on the same real audio), release the real dataset, and report a matched synthetic-versus-real performance comparison; alternatively, temper the claims accordingly. The limitation paragraph in §7 acknowledges TTS dependence but does not quantify the gap.","section":"§4, Tables 1–2, 5–7"},{"comment":"The scene embedding is underspecified: it is stated that a Minkowski Engine sparse ConvNet produces a 512-dimensional feature vector that is concatenated with audio features, but no training objective, loss, or usage in the downstream losses is described, and it is unclear whether the scene encoder is trained end-to-end or frozen. Without this information the method cannot be reproduced, and the contribution of the scene embedding to the reported results is unknown. Please specify the scene encoder's architecture, training procedure, and exact place in the pipeline.","section":"§3.2"},{"comment":"The loss weights λa, λb, λc in Eq. (4) and the Object Mention Detection threshold in §3.4 are not reported. These are free hyperparameters that directly affect the trained model and the set of relational objects used at inference. Their values are needed for reproducibility and to assess sensitivity; the paper should report them and ideally a small sensitivity analysis.","section":"Eq. (4), §3.4"},{"comment":"It is unclear whether the AP-Refer baseline (23.09 overall acc@0.25) is obtained by the authors using the identical Spark-TTS audio input or is taken from the original AP-Refer paper, which may have used a different TTS voice. If the number is from the original paper, the comparison in Table 1 is confounded by TTS differences. The text in §5.2 states \"Given the same audio input\" but does not describe how this was ensured. Although Table 5 (Matcha-TTS) suggests the margin survives a different TTS, the paper should state explicitly whether AP-Refer was re-run on the same audio, and if not, provide that result.","section":"Table 1"}],"minor_comments":[{"comment":"\"highlight the promise\" should be \"highlighting the promise\".","section":"Abstract"},{"comment":"\"scence\" should be \"scene\".","section":"§3.1"},{"comment":"\"embedd\" should be \"embed\".","section":"§3.2"},{"comment":"The table reports only a subset of metrics per dataset and leaves some cells as \"x\"; please indicate the sample counts for each subset and unify the metric reporting across the three datasets.","section":"Table 7"},{"comment":"The response-time comparison would be clearer if the hardware configuration and the definition of each reported time (e.g., whether user speaking time is included) were given explicitly.","section":"§6.3"},{"comment":"Appendices B–F consist largely of textbook ASR material (wav2vec 2.0, CTC, MFCC) that is not needed to understand or reproduce the method; I recommend trimming these to the relevant details or moving them to supplementary material.","section":"Appendices B–F"}],"recommendation":"major_revision","confidential_remarks":"The core technical contribution is sound on synthetic data, but the synthetic-to-real gap is the central uncertainty that should be resolved before publication. The appendices B–F appear disproportionately long and largely tutorial; I would encourage the editor to request a substantial trim. The real-world dataset withholding is acceptable during review but must be resolved at publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a legitimate advance in the small but real subfield of audio-based 3D visual grounding. The authors take the AP-Refer idea and make it work much better, with a sensible decomposition — object mention detection plus audio-guided attention — and solid gains on standard splits. The main caveat is that the headline numbers are all on synthetic TTS audio; the real-speech test is tiny, has no baselines, and isn’t released. That doesn’t sink the paper, but it caps how strong the central claim can be.\n\nWhat’s new: this is the second audio-based 3DVG method after AP-Refer, and it contributes two things AP-Refer lacks: an explicit multi-label object-mention detection head (which object categories are mentioned in the utterance) and audio-conditioned self/cross attention over object proposals. Those are meaningful architectural ideas, not just tweaks. The synthetic audio benchmarks for ScanRefer/Nr3D/Sr3D are useful resources if released.\n\nWhat’s done well: the ScanRefer comparison is clean — same audio input, standard metric, and a large margin over AP-Refer (40.02 vs 23.09 acc@0.25). They also report an Audio2Text pipeline so they’re not hiding behind weak text baselines. The ablation of the attention module is informative (4.6 points overall when swapped for an MLP). The limitation section is honest about dependence on TTS quality and segmentation.\n\nSoft spots, in proportion:\n- The synthetic-to-real transfer is the load-bearing assumption. All main training and evaluation use Spark-TTS audio; the only real-voice evidence is 549 samples, no baselines, unreleased. The drop from 40.02 to 36.72 on real ScanRefer suggests a gap exists but is not huge; without AP-Refer run on the same real data, the claimed advantage over AP-Refer is unverified in practice. This is fixable before final acceptance.\n- Missing details: loss weights λa/b/c, OMD threshold, and the scene embedding in §3.2 (no objective, no ablation, unclear downstream use). That’s sloppy but not fatal.\n- The Nr3D/Sr3D tables only compare against ReferIt3D and ScanRefer, both from 2020. More recent text methods would give a fairer picture of “competitive with text.”\n- No error bars, and the code/data are not yet released, which hampers independent verification.\n\nWho it’s for: researchers in 3D grounding, embodied AI, or speech-conditioned perception. It deserves a serious referee; the weaknesses are addressable in revision rather than structural. Recommendation: send to peer review, and ask for real-data baselines, the missing hyperparameters, an OMD ablation, and release of code and datasets.","headline":"Solid second-generation audio-based 3D grounding paper that clearly beats AP-Refer on synthetic audio, but the headline claim rests on Spark-TTS voices and the real-speech check is too thin to carry the practical claim.","tokens_in":32097,"tokens_out":3558,"would_cite":true,"duration_ms":40770,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Raw spoken descriptions, not typed text, can drive 3D visual grounding; the proposed Audio-3DVG pipeline reports the best audio-based accuracy to date on the ScanRefer, Sr3D, and Nr3D benchmarks and matches several text-based systems.","keywords":["audio-based 3D visual grounding","spoken language grounding","object mention detection","audio-guided attention","point cloud localization","wav2vec 2.0","text-to-speech synthesis","ScanRefer benchmark"],"falsifier":"Record a public test set of several thousand naturally spoken referring expressions over the same ScanRefer and ReferIt3D sentences, run the released Audio-3DVG model on this set without any fine-tuning, and compare its acc@0.25 against the text-based InstanceRefer number (37.69): if the real-speech accuracy falls below that text baseline or lands within a couple of points of the audio-only AP-Refer, the synthetic-to-real transfer assumption would be the point of failure.","tokens_in":1961,"feed_emoji":"🎤","tokens_out":2619,"duration_ms":95991,"temperature":0.7,"pith_summary":"This paper tries to establish that spoken language can serve as the primary input for 3D visual grounding, the task of picking out the object a description refers to inside a 3D point cloud of a room, without first transcribing the speech to text. The authors argue that treating voice as a second-class input behind typed text, or routing it through an automatic speech recognition step, adds latency and loses acoustic information, so they build a model that consumes raw audio directly. Their central results are a large jump over the previous audio-only system on ScanRefer (40.02 versus 23.09 accuracy at IoU 0.25) and parity or better relative to several text-based competitors, plus a 549-utterance real-voice test set for benchmarking. If the results hold up on real speech, hands-free and low-latency voice control of robots and assistants becomes a practical route into 3D scene understanding.","feed_headline":"Speech-only 3D grounding tops audio baselines, rivals text systems","feed_subtitle":"The model skips speech-to-text and still beats its text-based rival on the ScanRefer benchmark.","key_machinery":"The load-bearing mechanism is the Audio-Guided Attention Module, an attention layer in which the audio vector is injected directly into the query, key, and value projections of every object proposal: $q_i = W_q o_i + W_q^{(a)} a$, $k_j = W_k o_j + W_k^{(a)} a$, $v_j = W_v o_j + W_v^{(a)} a$, so that attention scores between objects are conditioned on what was said. It comes in two variants: audio-guided self-attention, which separates the target from same-class distractors, and audio-guided cross-attention, which links each target candidate to the relational objects flagged by the companion Object Mention Detection head, a multi-label classifier that predicts which object classes the utterance mentions. PointNet++ object embeddings, GloVe label embeddings, and normalized center and size coordinates are concatenated per proposal, and the whole system is trained with three losses: audio class classification, mention detection, and target-object classification.","core_discovery":"Audio-3DVG is a detection-then-matching pipeline: instance segmentation carves the scan into object proposals, a wav2vec 2.0 encoder turns the utterance into a fixed audio vector, and an audio class classifier, an Object Mention Detection head, and an Audio-Guided Attention module jointly select the referred object. The paper's claim is that decomposing the utterance into 'which category is the target' and 'which other object classes are mentioned' makes grounding substantially easier in cluttered scenes, because mentioned objects act as spatial anchors. On ScanRefer the framework reaches 40.02 acc@0.25 overall, 78.26 on the unique-object subset, and 32.78 acc@0.5, against 23.09 and 13.77 for the prior audio-only AP-Refer; grafting the attention module onto the text-based InstanceRefer baseline raises that baseline from 37.69 to 42.84, and on Nr3D and Sr3D the method reports 37.4 and 48.3 overall accuracy. The framework is trained and evaluated on text-to-speech voices synthesized from the standard ScanRefer, Sr3D, and Nr3D text benchmarks, with the 549-sample real-voice set used only as a final held-out test.","pith_inferences":["Because the head-to-head win over InstanceRefer (40.02 versus 37.69) is measured on TTS audio, a fair reading is that the method is state of the art for synthetic-speech grounding; whether the margin survives on real microphones remains open until the released real-voice set is used as a matched benchmark with baselines.","The same trick of adding a global modality vector to query, key, and value projections could transfer to other grounded-language tasks, such as 2D referring expression comprehension with spoken input or video grounding, wherever the query arrives as speech rather than text.","The real-voice test set of 549 utterances across nine speakers is smaller than the validation splits it is meant to represent; a released set of over a thousand utterances with AP-Refer and text-pipeline baselines run under identical settings would turn the reported real-world numbers into a stable benchmark."],"forward_implications":["End-to-end voice-to-object grounding skips automatic speech recognition, cutting measured pipeline time from 1,200 ms for a transcribe-then-ground system to 1,050 ms for the direct system.","Object Mention Detection transfers across modalities: adding the audio-guided attention module to the text-based InstanceRefer raises its ScanRefer overall accuracy from 37.69 to 42.84, suggesting relational-mention supervision helps regardless of input modality.","Synthesized speech is a scalable training resource: Spark-TTS and Matcha-TTS voices both train the same architecture to audio state of the art on ScanRefer (40.02 versus 39.14 overall acc@0.25), so the method does not depend on one specific TTS voice.","The audio classifier alone reaches 96% accuracy and mention detection F1 stays around 0.71-0.77, placing the remaining error in object reasoning rather than speech understanding.","Real-world voice grounding is tractable even at small scale: across 549 recorded utterances the model logs 36.72 acc@0.25 on a ScanRefer subset and 46.8 overall on Sr3D, with the dataset planned for release."],"supporting_citations":[{"why":"The prior audio-only 3D grounding system (AP-Refer) whose ScanRefer numbers (23.09 acc@0.25) define the baseline the method must beat.","marker":"[94]"},{"why":"ScanRefer, the primary benchmark that supplies the referring sentences, the 800 ScanNet scenes, and the official train/validation split used for the main evaluation.","marker":"[11]"},{"why":"ReferIt3D, the source of the Nr3D and Sr3D datasets and of the ground-truth-proposal evaluation protocol used for those two benchmarks.","marker":"[1]"},{"why":"wav2vec 2.0, the self-supervised speech encoder whose representations the entire audio branch is built on.","marker":"[6]"},{"why":"PointGroup, the instance segmentation backbone that produces the object proposals from the raw point cloud.","marker":"[43]"},{"why":"Spark-TTS, the text-to-speech model used to synthesize the audio descriptions that replace text input during training and the main evaluation.","marker":"[85]"},{"why":"InstanceRefer, the text-based two-stage grounding baseline that Audio-3DVG is compared against and that the attention module is ablated on.","marker":"[90]"},{"why":"Whisper, used to transcribe audio back into text so that text-based models can be evaluated on the same spoken inputs for a fairer comparison.","marker":"[74]"}],"fun_headline_variants":["Speech-only 3D grounding beats audio, rivals text","Audio-3DVG: new SOTA for 3D visual grounding from speech","Object-mention cues make audio 3D grounding SOTA","Speech-driven 3D grounding: mention detection lifts SOTA"],"cache_read_input_tokens":34176,"weakest_assumption_plain":"The training and headline evaluation assume that text-to-speech voices are faithful stand-ins for real spoken descriptions; every state-of-the-art number except the small 549-sample test is measured on those synthetic voices, and that test has no baselines and is not yet released.","fun_headline_variants_meta":{"raw":{"variants":["Speech-only 3D grounding beats audio, rivals text","Audio-3DVG: new SOTA for 3D visual grounding from speech","Object-mention cues make audio 3D grounding SOTA","Speech-driven 3D grounding: mention detection lifts SOTA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00058,"raw_usage":{"total_tokens":2785,"prompt_tokens":1049,"completion_tokens":1736,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":1661}},"tokens_in":665,"tokens_out":1736,"duration_ms":13876,"temperature":1.0,"reasoning_tokens":1661,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:09:33.903319+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record a public test set of several thousand naturally spoken referring expressions over the same ScanRefer and ReferIt3D sentences, run the released Audio-3DVG model on this set without any fine-tuning, and compare its acc@0.25 against the text-based InstanceRefer number (37.69): if the real-speech accuracy falls below that text baseline or lands within a couple of points of the audio-only AP-Refer, the synthetic-to-real transfer assumption would be the point of failure.","supporting_citations":[{"cited_title":"3d visual grounding-audio: 3d scene object detection based on audio","cited_arxiv_id":null,"evidence_quote":"The prior audio-only 3D grounding system (AP-Refer) whose ScanRefer numbers (23.09 acc@0.25) define the baseline the method must beat."},{"cited_title":"Pointgroup: Dual-set point grouping for 3d instance segmentation","cited_arxiv_id":null,"evidence_quote":"PointGroup, the instance segmentation backbone that produces the object proposals from the raw point cloud."},{"cited_title":"Spark-tts: An efficient llm-based text-to-speech model with single-stream decoupled speech tokens","cited_arxiv_id":null,"evidence_quote":"Spark-TTS, the text-to-speech model used to synthesize the audio descriptions that replace text input during training and the main evaluation."},{"cited_title":"Instancere- fer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring","cited_arxiv_id":null,"evidence_quote":"InstanceRefer, the text-based two-stage grounding baseline that Audio-3DVG is compared against and that the attention module is ablated on."},{"cited_title":"Robust speech recognition via large-scale weak supervision","cited_arxiv_id":null,"evidence_quote":"Whisper, used to transcribe audio back into text so that text-based models can be evaluated on the same spoken inputs for a fairer comparison."}],"review_version":1}