{"id":"4b11216e-d1eb-4218-b268-bf53af585ca6","arxiv_id":"2611.16787","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"MESH evaluates spatial speech understanding across egocentric and allocentric views in multi-receiver environments, finding that current models struggle to associate speech with the correct physical source.","lead":"MESH is a new benchmark that tests whether AI systems can correctly associate speech with the right speaker and location in multi-receiver, moving acoustic scenes. It reveals that current models fail badly at this task, with best accuracies below 42% and word error rates above 59%, highlighting a major gap between having distributed audio data and using it effectively.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Simulated scenes lack ecological validity for the claim that current models cannot effectively use distributed acoustic information; the simulation-to-reality gap is unaddressed.","rationale":"The reader's weakest_assumption correctly identifies the simulation-to-reality transfer as the weakest point. The paper's central claim is about a fundamental limitation of current models, but the evidence is entirely from simulated data. The authors acknowledge this limitation, but the claim's strength depends on the assumption that simulation is representative. The concrete test I propose is feasible (small-scale real recording) and would directly address the concern. I recommend CONDITIONAL acceptance: the paper should be accepted if the authors either provide real-world validation or substantially weaken the claim to be about simulated environments only. The reader's verdict was ACCEPT with HIGH confidence; I think the confidence should be lower given this unresolved concern. The paper is otherwise strong, but the central claim's generality is not fully supported.","tokens_in":23858,"tokens_out":617,"duration_ms":6567,"concrete_test":"Record a small set of real multi-receiver scenes (e.g., 10 rooms, 2-3 receivers, 1-2 moving speakers) with known ground truth (e.g., via motion capture and close-talk mics). Evaluate the best-performing model(s) from the paper (e.g., Qwen3.8-Omni-Flash, MESH-agent) on these real recordings using the same task definitions. If performance drops significantly (e.g., >20% relative degradation in accuracy or WER increase >10% absolute) compared to simulation, the simulation-to-reality gap is a critical concern. If performance is comparable, the concern is mitigated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that current models cannot effectively use distributed acoustic information for spatial speech understanding—rests entirely on simulated acoustic scenes (SoundSpaces 2.0 FOA rendering, horizontal motion, fixed height, ideal self-playback removal, synchronized recordings, known poses). The authors acknowledge this limitation (Limitations: 'Generalization to spontaneous conversations, other languages, and physical recording environments remains untested.'). However, the claim is not merely about simulation performance; it is about a fundamental gap in capability. If real-world conditions (clock drift, calibration errors, missing recordings, asynchronous streams, non-ideal self-playback removal, 3D motion, varying microphone types) substantially change the difficulty or the relative model rankings, then the conclusion that 'existing models are not yet able to effectively leverage peer information' may not hold in practice. The paper does not provide any evidence—even a small-scale real-world recording or a perturbation analysis—that the simulation findings are robust to these real-world factors. The weakest assumption is that the simulation is representative enough to support a general claim about model capability. This is a load-bearing concern because the entire benchmark's validity for real-world deployment depends on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper introduces MESH, a benchmark for evaluating spatial speech understanding in multi-receiver environments. It defines egocentric and allocentric views, constructs ~40,000 questions across 13 static/dynamic localization and transcription tasks using simulated acoustic scenes (SoundSpaces 2.0), and evaluates 11 audio-model configurations plus an agentic system. Results show poor performance, especially in distance estimation and multi-speaker scenarios, and limited benefit from peer information and distributed recordings.","tokens_in":24230,"tokens_out":1645,"duration_ms":12119,"significance":"If the benchmark is valid, it provides a systematic evaluation of a relatively underexplored capability: associating speech with the correct physical source across receivers and motion. The benchmark design is thorough, with automated checks and manual validation. The results consistently show that current models struggle, which could guide future research. However, the significance is tempered by the simulation-to-reality gap and the acknowledged limitations.","major_comments":[{"comment":"The central claim that current models cannot effectively use distributed acoustic information rests entirely on simulated scenes with ideal assumptions (synchronized recordings, known poses, ideal self-playback removal, horizontal motion, fixed height). The authors acknowledge this limitation (lines [b186], [b191]), but the claim is about a fundamental capability gap, not just simulation performance. Real-world factors (clock drift, calibration errors, missing recordings, asynchronous streams, non-ideal self-playback removal, 3D motion, varying microphone types) could substantially change difficulty or relative model rankings. The paper provides no evidence—even a small-scale real-world recording or perturbation analysis—that the simulation findings are robust to these factors. This is a load-bearing concern because the benchmark's validity for real-world deployment depends on it. The authors should either temper the claim to be about simulated environments or provide evidence of robustness.","section":"Limitations (Observation assumptions)","authors_acknowledge":"Generalization to spontaneous conver- sations, other languages, and physical recording environments remains untested.","authors_acknowledge_p":0.86},{"comment":"The ablation study on peer information and distributed recordings is conducted on a subset (100 rooms, 5,090 questions) and uses only three models. The authors acknowledge that removing metadata also changes prompt length, which is a confound (line [b385]). The results show inconsistent effects, and the sample size may be insufficient to draw strong conclusions. The claim that 'peer information provides limited benefits' and 'distributed recordings offer selective gains' is supported by the data, but the analysis is not comprehensive enough to rule out that better integration methods could yield consistent improvements. The authors should discuss this limitation more explicitly.","section":"§4.3 / Appendix B.3 (Controlled information comparisons)","authors_acknowledge":"Shorter prompts are a confound; no length-matched placebo is included.","authors_acknowledge_p":0.78},{"comment":"The Overall score (Eq. 4) weights localization and transcription equally within each view, but the composition of tasks differs between views (8 egocentric vs. 5 allocentric). The score is also sensitive to the threshold choice (20° direction error, 20% distance error, 2 m position error). While the authors report task-level metrics separately, the Overall score is used for summary comparisons. The authors should clarify whether the Overall score is intended as a primary metric or just an illustrative aggregate, and discuss its limitations.","section":"§4.2 / Table 2 (Egocentric results)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The manual validation covers only 57 scene-question reviews (36 pilot, 21 formal) out of ~40,000 questions, with no inter-rater agreement established (lines [b190], [b210]). The authors acknowledge this limitation. However, the benchmark is presented as a reliable evaluation tool. The limited validation scope means that undetected annotation errors could affect results. The authors should either expand validation or explicitly state that the benchmark is preliminary and may contain errors.","section":"§3.4 / Appendix C.2 (Data curation pipeline)","authors_acknowledge":"The preliminary listening audit covers only a small subset of scenes and does not establish cross-receiver consistency.","authors_acknowledge_p":0.95}],"minor_comments":[{"comment":"The abstract states 'nearly 40,000 questions' but the main text says 'approximately 40,000 questions' (line [b33]). This is consistent, but the exact count should be specified for reproducibility.","section":"Abstract","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The introduction mentions 'an exhibition hall' as a concrete example, but the benchmark uses simulated exhibition rooms. This could be clarified to avoid implying real-world validation.","section":"§1 (Introduction)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The definition of egocentric view uses Eq. (1) for relative position, but the paper does not specify how orientation is handled for direction queries (azimuth/elevation). This is clear from context but could be explicitly stated.","section":"§3.1 (Scene and Coordinate Views)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The tables use abbreviations like 'Acc↑' and 'RelAcc↑' without defining them in the caption. They are defined in §4.1, but a brief reminder in the caption would improve readability.","section":"Table 2 and Table 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The threshold for direction error is 20°, for distance error 20% relative, and for position error 2 m. These thresholds are reasonable but arbitrary. The authors should discuss sensitivity to threshold choice or provide additional metrics (e.g., mean absolute error).","section":"§4.1 (Evaluation metrics)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The definition of coverage and handling of missing predictions is clear, but the diagnostic error assignment (180° for missing directions) could be mentioned in the main text for transparency.","section":"Appendix B.1 (Metrics and Reporting Scope)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The figure shows WER vs. human sources, SNR, and SIR. The connecting lines between conditions are described as 'ordered conditions, not a continuous numerical interval to Clean' (line [b363]). This is a minor clarity issue; the authors could use bar charts instead of line plots to avoid implying interpolation.","section":"Figure 2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The listening audit is described in detail, but the sample size (57 reviews) is very small. The authors acknowledge this, but the inclusion of this audit in the main paper (via reference) might give a false impression of thorough validation. Consider moving to supplementary material or clearly stating its preliminary nature.","section":"Appendix C.4 (Human Review Interface and Procedure)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Some references are to arXiv preprints (e.g., [Biswas et al., 2026], [Hu et al., 2026], [Kumar et al., 2026]). This is acceptable for a rapidly evolving field, but the authors should ensure that cited results are stable and not subject to major revision.","section":"References","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper is generally well-written and the benchmark design is thorough. The code and dataset release is commendable and supports reproducibility.","section":"General","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This is a solid benchmark paper. The core contribution is MESH, a dataset and evaluation protocol that combines multi-receiver FOA recordings, receiver motion, numerical trajectories, and spatially cued transcription with physical error metrics. No existing benchmark does all of that together. The egocentric/allocentric view distinction is a useful conceptual framing, and the 13 tasks cover a sensible range of static and dynamic localization and transcription problems. The construction pipeline is well-documented, with automated checks and manual validation. The authors evaluate eleven model configurations and an agentic system, and the results are informative: current models perform poorly on associating speech with the correct physical source, especially in crowded and dynamic scenes. The paper is honest about its limitations—simulated data, English only, horizontal motion, ideal assumptions—and the dataset and code are released.\n\nThe main soft spot is the simulation-to-reality gap. The central claim that current models cannot effectively use distributed acoustic information rests entirely on simulated scenes with synchronized recordings, known poses, ideal self-playback removal, and no clock drift or calibration errors. The authors acknowledge this, but they do not provide any evidence—not even a small-scale real-world recording or a perturbation analysis—that the findings are robust to real-world factors. This is a load-bearing concern if the benchmark is meant to guide real-world deployment. However, the paper is careful to frame the results as highlighting a gap in controlled conditions, and the benchmark is still valuable for comparing models under those conditions. The stress-test note overstates this concern: the paper does not claim real-world validity, and the limitations section is explicit. The benchmark is a useful tool for measuring progress in a controlled setting, not a deployment-ready evaluation.\n\nMinor issues: the MESH-agent evaluation used a release with prior results rather than an untouched test set, and the contribution of agentic planning is not isolated. The confidence intervals do not account for multiple comparisons. These are acknowledged and are minor.\n\nWho this is for: researchers working on spatial audio understanding, multi-receiver speech processing, or embodied listening. The paper deserves a serious referee. I would accept peer review.\n\nRecommendation: Accept with minor revisions. The authors should add a small-scale real-world validation or at least a perturbation analysis to address the simulation-to-reality gap, but the paper is already a solid contribution as is.","headline":"A thorough, well-constructed benchmark for multi-receiver spatial speech understanding; the simulation-to-reality gap is real but the paper acknowledges it, and the benchmark is still valuable for controlled comparisons.","tokens_in":24652,"tokens_out":572,"duration_ms":5443},"pith_extraction":{"msc":[],"pacs":["43.60.Jn","43.71.Ky","43.66.Pn"],"model":"deepseek/deepseek-v4-flash","headline":"MESH benchmark reveals current AI systems cannot reliably associate speech with the correct physical source in multi-receiver environments.","keywords":["spatial audio","benchmark","multi-receiver","speech understanding","localization","transcription","egocentric","allocentric"],"falsifier":"If a model achieved near-human performance on MESH (e.g., >90% localization accuracy and <10% WER on spatial transcription) using the same input protocols, the paper's central claim that current systems cannot effectively use distributed acoustic information would be falsified for that model. Alternatively, if a physical recording experiment showed that human listeners or a baseline system perform similarly poorly on analogous tasks in real rooms, the claim that the benchmark reveals a model-specific limitation rather than a fundamental difficulty of the task would be weakened.","tokens_in":25428,"feed_emoji":"🎤","tokens_out":1516,"duration_ms":11281,"temperature":0.7,"pith_summary":"The paper introduces MESH, a benchmark for evaluating spatial speech understanding in multi-receiver environments. It assesses whether AI systems can associate speech with the correct physical source across both egocentric (receiver-relative) and allocentric (world-frame) views. The authors test eleven audio-model configurations and a language-model agent on nearly 40,000 questions spanning static and dynamic localization and transcription tasks. The central finding is that current systems perform poorly: the best static distance estimation accuracy is only 21.90%, the best dynamic direction tracking is 41.54%, and word error rates for spatial transcription range from 59.15% to 73.10%. Ablation studies show that providing peer metadata and distributed recordings yields only selective and inconsistent improvements, underscoring a fundamental gap between access to distributed acoustic information and its effective use for spatial hearing.","feed_headline":"MESH: AI fails to match speech to speaker in crowds","feed_subtitle":"New benchmark shows best models get distance wrong 78% of the time and often transcribe the wrong person.","key_machinery":"The benchmark itself is the key machinery: a controlled dataset of simulated exhibition-room acoustic scenes with up to three human speakers and three robot receivers, each recording synchronized first-order Ambisonic audio. The central design is the separation of egocentric and allocentric views, which allows testing whether models can associate acoustic observations with physical sources in both receiver-relative and world coordinates. The 13 task definitions (e.g., ES-CDir, AD-Trans) specify exactly what cues are provided (transcripts, spatial anchors, regions) and what must be predicted (direction, distance, position, transcript), enabling precise diagnosis of which aspects of spatial hearing fail.","core_discovery":"The paper establishes that existing audio-language models and agentic systems cannot reliably associate speech with the correct physical source in multi-receiver acoustic scenes, particularly when multiple speakers are present or when sources or receivers move. The benchmark systematically tests this association through 13 tasks that require either localizing a speaker identified by transcript or transcribing speech identified by spatial cues, in both static and dynamic settings. The results show that direction estimates can be moderately accurate (up to 50% for transcript-conditioned direction) but distance estimates remain poor (below 22%), and spatial transcription frequently recovers speech from the wrong person (25% of positive answers match another human's speech). The paper argues that this reveals a fundamental gap between having access to distributed acoustic information and being able to use it effectively.","pith_inferences":["The failure to use peer metadata and distributed recordings suggests that current architectures lack a mechanism for cross-receiver feature alignment or geometric reasoning; this points toward a need for explicit 3D spatial representations rather than implicit learned embeddings.","The gap between direction and distance accuracy implies that models may be exploiting the strong directional cues available in first-order Ambisonics (which encode pressure gradients) but not the weaker distance cues (intensity, direct-to-reverberant ratio), suggesting that training on distance-related tasks or incorporating binaural cues could help.","The benchmark's simulated nature (horizontal motion, fixed height, ideal self-playback removal) means that real-world deployment would likely reveal even larger gaps, especially with unknown receiver poses, clock drift, and non-ideal microphone arrays."],"forward_implications":["Future spatial audio models must explicitly solve the speech-source association problem rather than treating localization and transcription as separate tasks.","Peer information and distributed recordings, while conceptually useful, require new architectural mechanisms to be reliably exploited; current models show no consistent benefit from them.","The finding that distance estimation and dynamic tracking are far weaker than direction estimation suggests that models rely on interaural cues but fail to integrate intensity, reverberation, or motion cues.","The high rate of wrong-speaker transcription (25% of positive answers) indicates that spatial attention mechanisms in current audio-language models are not sufficiently selective.","The benchmark provides a standardized evaluation protocol that can drive progress in collaborative and distributed spatial hearing, analogous to how ImageNet drove visual recognition."],"supporting_citations":[{"why":"SoundSpaces 2.0 is used as the acoustic simulation platform for rendering first-order Ambisonic audio through room impulse responses.","marker":"Chen et al., 2022"},{"why":"LibriSpeech provides the 300-speaker speech inventory used for human speech and robot playback in the benchmark scenes.","marker":"Panayotov et al., 2015"},{"why":"JAEGER is one of the evaluated audio-model configurations, representing joint 3D audio-visual grounding and reasoning.","marker":"Liu et al., 2026a"},{"why":"Spatial-Omni is one of the evaluated audio-model configurations, providing spatial audio understanding via FOA encoding.","marker":"Zhu et al., 2026"},{"why":"SEA-LM is one of the evaluated audio-model configurations, focusing on egocentric spatial audio understanding for wearable microphone arrays.","marker":"Kumar et al., 2026"},{"why":"Qwen3-Omni is one of the evaluated audio-model configurations, representing a single-channel encoding LALM.","marker":"Xu et al., 2025"},{"why":"Qwen3-14B is used as the frozen planner for the MESH-agent system.","marker":"Yang et al., 2025a"}],"fun_headline_variants":["AI can't tell who spoke where in crowded rooms","New audio benchmark shows models fail at speaker tracking","MESH: Why AI can't match speech to the right person","Spatial hearing AI fails 78% of distance tests","Multi-receiver speech model still mixes up speakers"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The simulated acoustic scenes using SoundSpaces 2.0 with first-order Ambisonic rendering, horizontal motion, fixed height, and ideal self-playback removal are representative enough of real-world multi-receiver environments to support the general conclusion that current models cannot effectively use distributed acoustic information for spatial speech understanding.","fun_headline_variants_meta":{"raw":{"variants":["AI can't tell who spoke where in crowded rooms","New audio benchmark shows models fail at speaker tracking","MESH: Why AI can't match speech to the right person","Spatial hearing AI fails 78% of distance tests","Multi-receiver speech model still mixes up speakers"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000105,"raw_usage":{"cost":0.000105105,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000105105,"upstream_inference_prompt_cost":9.2547e-05,"upstream_inference_completions_cost":1.2558e-05},"total_tokens":1086,"prompt_tokens":1017,"completion_tokens":69,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":0}},"tokens_in":1017,"tokens_out":69,"cost_source":"openrouter_usage","duration_ms":2127,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":0,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T12:57:06.737482+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"If a model achieved near-human performance on MESH (e.g., >90% localization accuracy and <10% WER on spatial transcription) using the same input protocols, the paper's central claim that current systems cannot effectively use distributed acoustic information would be falsified for that model. Alternatively, if a physical recording experiment showed that human listeners or a baseline system perform similarly poorly on analogous tasks in real rooms, the claim that the benchmark reveals a model-specific limitation rather than a fundamental difficulty of the task would be weakened.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}