What's in the Box? Reasoning about Unseen Objects from Multimodal Cues

Alicia Zhang; Daniel Xu; Joshua B. Tenenbaum; Katherine M. Collins; Lance Ying; Max H. Siegel

REVIEW 3 major objections 5 minor 23 references

What's in the Box? Reasoning about Unseen Objects from Multimodal Cues

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A neurosymbolic model parses audio, vision, and language with neural networks and combines them with a Bayesian engine, judging which unseen objects are in which box with $r=0.78$ agreement with humans, while unimodal and vision-language…

desk verdict Useful new benchmark, but the Bayesian claim is heuristic and the headline correlations are statistically ungrounded. read the letter →

arxiv 2506.14212 v1 pith:22JAGQC4 submitted 2025-06-17 cs.AI

Lance Ying , Daniel Xu , Alicia Zhang , Katherine M. Collins , Max H. Siegel , Joshua B. Tenenbaum This is my paper

classification cs.AI

keywords neurosymbolicmodelmultimodalreasoningBayesianinferenceaudio-visualintegrationhiddenobjectWhat'sintheBoxhumanjudgmentcorrelationvision-language

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to establish that human-like reasoning about objects hidden from view can be produced by a neurosymbolic pipeline: neural networks convert sound, video, and a list of object names into structured descriptions, and a Bayesian engine then scores every way the objects could be placed inside the boxes. On a new 'What's in the Box?' task, the full model's graded judgments correlate with human judgments at $r=0.78$, while audio-only and vision-only versions fall to $r=0.55$ and $r=0.52$, and a large vision-language model baseline drops to $r=0.31$. The result matters because it suggests that monolithic neural models, however large, do not currently capture the joint, constraint-based inference humans perform across senses, whereas even modest neural parsers followed by structured probabilistic reasoning can. The paper also argues that this integration is not a simple average of the modalities: the joint posterior over complete placements behaves differently from either unimodal judgment.

What carries the argument

The load-bearing mechanism is the Bayesian hypothesis evaluator. It defines the state space as every ordered partition of the $N$ object names into $K$ boxes, $|H|=K!S(N,K)$, assigns a uniform prior, and writes the joint likelihood as $P(O,A|H)\propto \prod_i P(O_i|H_i)\,P(H_i|A_i)$, treating audio and visual evidence as conditionally independent. The visual factor $P(O_i|H_i)$ is computed by rejection sampling: drawing object and box dimensions from normal distributions and checking fit 1000 times. The audio factor is obtained by substituting the audio classifier's posterior $P(H_i|A_i)$ for the unavailable likelihood $P(A_i|H_i)$, a shortcut the paper acknowledges is not derived from a generative audio model. The posterior over hypotheses is then marginalized to produce per-object, per-box probability ratings.

What would settle it

A controlled replication could record each object-box configuration several times, estimate the true audio likelihood $P(A_i|H_i)$ empirically, and re-run the model with that likelihood in place of the classifier posterior; if placement rankings shift in cases where human judgments stay stable, the reported correlation depends on the substitution rather than on Bayesian integration itself.

Watch

Extended reading notes

Core claim

Using the 'What's in the Box?' game, the paper claims that combining neural perception with explicit Bayesian inference produces human-like graded beliefs about occluded objects. Given $N$ objects and $K$ boxes, the model enumerates all $K!S(N,K)$ placement hypotheses, assumes a uniform prior, and updates each hypothesis with $P(O,A|H)$, factored as a product over boxes of a visual fit term and an audio term. The visual fit comes from rejection-sampling object and box dimensions under normal distributions to check fit 1000 times; the audio term is obtained by querying the audio classifier with the sound of each box and candidate object labels. The authors report that this full model matches human probability ratings with $r=0.78$, that removing either modality lowers the correlation to roughly $0.5$, and that the vision-language baseline given the same video and instructions reaches only $r=0.31$. Qualitative examples show the full model resolving cases where vision and audio individually disagree, such as a yoga mat constrained by box size plus a laptop identified by collision sounds, and coins located by jingling.

Load-bearing premise

The inference engine's output is only as valid as the shortcut that replaces the audio likelihood $P(A_i|H_i)$ with the audio classifier's posterior $P(H_i|A_i)$, combined with the assumption that sound and visual fit are conditionally independent.

Editorial extensions

If this is right

Audio alone and vision alone in the same architecture both lose about a quarter of the correlation with human judgments, so the integration step, not any single parser, is doing the work.
A monolithic vision-language model given identical video and instructions trails the neurosymbolic model by a large margin, implying that large pretrained multimodal models are not yet a substitute for structured joint inference on ambiguous physical scenes.
The full model's judgments are not a weighted average of unimodal outputs, since joint inference over whole placements produces different marginals than either modality alone.
The hypothesis space grows combinatorially as $K!S(N,K)$, so applying the approach to more boxes or more objects will require approximate inference over placements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

One step beyond the paper: if the classifier-posterior-for-likelihood substitution is as fragile as it sounds, the model should degrade sharply when the audio classifier is miscalibrated or when two candidate objects make similar sounds; a calibration study on near-confusable object sets would show where the shortcut holds.
The conditional-independence approximation will likely break in richer scenes where one physical shake generates both the sound and the visible box motion; adding learned reliability weights per modality, which the paper lists as a limitation, would turn this approximation into a testable model of adaptive cue weighting.
The task itself could serve as a compact behavioral probe for physical reasoning in vision-language models: varying box-size constraints and object counts changes the difficulty of second-order reasoning such as two objects sharing a box, making it easy to test whether future foundation models close the gap.
In robotics, the same pipeline could maintain a belief distribution over sealed container contents from a single shake, a use the paper mentions only as a future direction.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, the useful thing: WiTB is a clean new benchmark, and the pipeline—LLM-generated object priors, CLAP audio classification, Gemini visual parsing, and a combinatoric hypothesis evaluation—is a reasonable engineering recipe for a hard multimodal reasoning task. The qualitative examples are genuinely illustrative, and the comparison against a monolithic VLM baseline is the right kind of baseline to include. Credit where due: the authors are upfront that the audio likelihood is replaced by the classifier's posterior.

Now the problems. The inference equation is not Bayesian as written. Equation (1) substitutes P(H|A) for P(A|H), and the paper explicitly admits this 'manipulation.' That substitution is not a harmless approximation; it changes the meaning of the update. The product of a likelihood from vision and a posterior from audio is a heuristic score, not a posterior. Conditional independence of audio and visual observations is also asserted without support. These are legitimate concerns, but note that the correlation with human judgment doesn't depend on the update being Bayesian in the strict sense—the model could still be predictive. So I'd treat this as a framing/justification flaw, not necessarily the thing that makes the empirical claim false.

More load-bearing is the stimulus exclusion. The authors built 25 stimuli and then dropped 5 because human split-half agreement was below 0.8. That is post hoc filtering on the outcome variable, and with 20 stimuli left the difference between r=0.78 and r=0.55/0.52/0.31 is reported without confidence intervals, significance tests, or a full-data analysis. The figure caption mentions 95% CIs, but the text gives no numbers. This selective reporting directly undermines the headline comparison.

Also, no code or data is released, which makes it hard to check the parsing, the priors, or the correlations.

Who is this for? People interested in neurosymbolic reasoning, multimodal perception, and new benchmarks for human-like inference. The WiTB task itself is a useful addition to the toolkit. The model as described is not a validated Bayesian account, and the empirical advantage is not statistically supported as presented. But those are fixable with a revision: reframe the update as a heuristic or derive a proper likelihood, justify or remove the exclusion rule, add significance testing, and release the stimuli.

Recommendation: send to peer review. The benchmark deserves referee time, and the flaws, while significant, are addressable. A serious referee should push for the above rather than a desk reject.

Referee Report

3 major / 5 minor

Summary. The paper presents a neurosymbolic model for reasoning about hidden objects in boxes from multimodal cues. The model parses language, vision, and audio into structured representations using LLMs, VLMs, and CLAP, then performs Bayesian inference over all possible object-placement hypotheses. The authors introduce the 'What's in the Box?' task, collect human judgments from 54 participants on 25 stimuli (with 5 excluded), and report that their full model correlates with human judgments at r=0.78, outperforming audio-only (0.55), vision-only (0.52), and Gemini 2.0 Flash (0.31) baselines. They argue that structured Bayesian integration of neural perceptual outputs yields more human-like inference than monolithic multimodal models.

Significance. If the results hold, the paper would provide a useful demonstration that neurosymbolic inference over neural perceptual outputs can capture human multimodal reasoning in an open-ended task, and the WiTB paradigm could be a valuable benchmark. Strengths include a clear architectural decomposition, no fitting of the model to human judgments (so the correlation is not circular in the standard sense), a novel stimulus set, and qualitative analyses illustrating the complementary roles of audio and visual cues. The main weaknesses are statistical: the headline correlation advantage rests on post hoc stimulus exclusion and point estimates without confidence intervals, and the Bayesian derivation contains an unjustified substitution of a classifier posterior for a likelihood.

major comments (3)

[Generating and evaluating hypotheses, Eq. (1)-(2)] Equation (1) replaces the audio likelihood P(A|H) with the posterior P(H|A), which is only valid up to a constant if the prior over hypotheses is uniform and the observation A is fixed; the subsequent factorization in Eq. (2) is more problematic. The notation conflates the hypothesis index i with the box index: P(H_i|A_i) in the product is not well-defined because H_i is a full placement hypothesis while A_i is the audio of a single box. Moreover, P(H_i^n|A_i) is computed as the product of per-object classifier posteriors ∏_{o∈H_i^n} P(o|A_i), which assumes that the presence of objects within a box is conditionally independent given the box audio and penalizes boxes containing more objects. This is a load-bearing heuristic for the model's posterior, and the paper provides no generative justification or validation. The authors should either derive a proper likelihood P(A_i|contents) or demonstrate that this approximation does not drive the reported correlation.
[Experiment and Results] The exclusion of 5 of 25 stimuli based on a split-half correlation of human judgments below 0.8 is post hoc selection on the outcome variable. No results for the full stimulus set or a sensitivity analysis are reported, so the headline r=0.78 versus the baseline correlations could be an artifact of favorable stimulus selection. In addition, the Results section reports only point estimates; there are no confidence intervals for the correlations and no significance tests for the differences between the full model and the unimodal or VLM baselines, despite the Figure 3 caption claiming a 'significantly better fit.' The authors should report the full-data correlations, bootstrap or Fisher-transformed CIs, and a formal comparison of dependent correlations (e.g., Steiger's test or a permutation test).
[Audio component; Discussion, Error Analysis] The CLAP model's posterior P(o|A_i) is used as the audio evidence term without calibration or validation. The audio of a shaken box is a joint acoustic mixture of all objects inside, so the per-object probabilities output by CLAP for candidate labels are not the same as the likelihood of observing that audio given a particular set of contents. The error analysis notes that CLAP misses nuanced sounds in mixtures, but the model's scores are treated as probabilities in the Bayesian update. This is closely related to Major Comment 1, and it should be addressed either by reframing the model as a heuristic scoring function rather than a Bayesian model, or by providing evidence that the uncalibrated posteriors behave like likelihoods in this task.

minor comments (5)

[Abstract and References] There are several typographical errors: 'Keywords:perception' is missing a space, 'Ernst, M. ) (2007)' has a stray parenthesis, and the Alais & Burr reference has a doubled closing parenthesis.
[Experiment] The sentence 'where the of scale for each item automatically sums to 100' appears to be missing a word; it should likely read 'where the value of the scale for each item automatically sums to 100.'
[Figure 3] The caption states 'Error bars show standard error and CI indicates 95% confidence interval,' but the figure does not clearly distinguish which visual elements correspond to standard error versus confidence intervals; please clarify the plotting convention.
[General] The paper does not state whether stimuli, human data, or code will be made available; for reproducibility, please add a data/code availability statement.
[Computational Model, Eq. (1)] The notation in Eq. (1) is inconsistent: H_i denotes both a hypothesis and a box index; please use distinct subscripts for boxes (e.g., n) and hypotheses (e.g., i) throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the full model is evaluated against external human judgments and is not fitted to them.

full rationale

The derivation chain is self-contained with respect to the human-judgment benchmark. The model's perceptual modules (Llama 3.1 70B, CLAP, Gemini 2.0 Flash) are pretrained and produce fixed symbolic or posterior outputs, and the Bayesian engine in Equation (1) has no parameters fitted to the WiTB human ratings. The reported correlation r=0.78 is therefore an external evaluation, not a quantity reconstructed from the model's inputs. The self-citations (e.g., 'Similar to prior work on neurosymbolic reasoning (Hsu et al., 2023; Wong et al., 2023; Ying et al., 2023)') are used as general framing rather than as load-bearing justification, and no uniqueness theorem or prior fitted result is invoked to force the model's conclusions. The acknowledged substitution of P(H|A) for P(A|H) and the conditional-independence assumption make the inference non-Bayesian in a strict mathematical sense, but they do not make the prediction equivalent to its inputs. The post hoc exclusion of five low-agreement stimuli is a statistical reporting concern that warrants caution about the point estimates, but it does not create a definitional identity between the model's outputs and human judgments. Overall, no circular step is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The model introduces no new physical or mathematical entities. Its central parameters are the LLM-generated object priors and the post hoc stimulus exclusion rule, both of which are under-specified. The main conceptual issue is the unjustified use of a posterior as a likelihood.

free parameters (2)

Object attribute priors (size, weight, material, rigidity) generated by Llama 3.1 70B = not specified (per-object JSON values)
These priors directly determine the rejection-sampling fit probabilities, and the LLM is prompted without calibration to real object statistics.
Stimulus exclusion threshold = split-half correlation < 0.8
Five stimuli were excluded post hoc because human agreement was below this threshold, which affects the reported correlation.

assumptions (4)

domain assumption Audio and visual observations are conditionally independent given a placement hypothesis.
Invoked in Eq. (1) to factor the joint likelihood into P(O|H) and P(H|A); no empirical or theoretical justification is provided.
ad hoc to paper The audio classifier's posterior P(o|A_i) can be used directly as a factor in the likelihood term P(H_i|A_i).
The paper states that the audio likelihood is hard to estimate, so it replaces it with the classifier's posterior. This is not a standard Bayesian step and changes the meaning of the posterior.
domain assumption The LLM-generated object dimensions and uncertainties adequately represent the true physical variability of the objects.
Used to sample object and box dimensions during rejection sampling; no validation against real measurements is reported.
standard math The uniform prior over object placements is appropriate.
Assumed in the hypothesis-generation step; this is a simple default and not a major source of error.

how reviews work

0 comments

Cite this review

Pith. "Pith review of What's in the Box? Reasoning about Unseen Objects from Multimodal Cues." pith.science (2026). https://pith.science/paper/22JAGQC4

@misc{pith2026250614212,
  author       = {Pith},
  title        = {Pith review of: What's in the Box? Reasoning about Unseen Objects from Multimodal Cues},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/22JAGQC4}},
  note         = {Machine review of arXiv:2506.14212}
}

read the original abstract

People regularly make inferences about objects in the world that they cannot see by flexibly integrating information from multiple sources: auditory and visual cues, language, and our prior beliefs and knowledge about the scene. How are we able to so flexibly integrate many sources of information to make sense of the world around us, even if we have no direct knowledge? In this work, we propose a neurosymbolic model that uses neural networks to parse open-ended multimodal inputs and then applies a Bayesian model to integrate different sources of information to evaluate different hypotheses. We evaluate our model with a novel object guessing game called ``What's in the Box?'' where humans and models watch a video clip of an experimenter shaking boxes and then try to guess the objects inside the boxes. Through a human experiment, we show that our model correlates strongly with human judgments, whereas unimodal ablated models and large multimodal neural model baselines show poor correlation.

Figures

Figures reproduced from arXiv: 2506.14212 by the authors.

**Figure 2.** Our neurosymbolic model. (a) The model first uses neural networks to parse multimodal input to a structured JSON [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Correlation plots comparing belief judgments from humans (y-axis) against models (x-axis). Each dot represents a [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Two qualitative examples comparing model and human ratings on the location of the objects. The bars represent [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 18 canonical work pages

[1]

David, B., Finn, C., Fu, C., Gopalakrishnan, K., Haus- man, K., et al. (2022). Do as I can, not as I say: Grounding language in robotic affordances.arXiv preprint arXiv:2204.01691

arXiv 2022
[2]

Alais, D., & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration.Current biology, 14(3), 257–262

work page 2004
[3]

W., Hamrick, J

Battaglia, P. W., Hamrick, J. B., & Tenenbaum, J. B. (2013). Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences,110(45), 18327–18332

work page 2013
[4]

W., Jacobs, R

Battaglia, P. W., Jacobs, R. A., & Aslin, R. N. (2003). Bayesian integration of visual and auditory signals for spa- tial localization.Josa a,20(7), 1391–1397

work page 2003
[5]

A., & Wang, H

Elizalde, B., Deshmukh, S., Ismail, M. A., & Wang, H. (2022). Clap: Learning audio concepts from natural lan- guage supervision

work page 2022
[6]

) (2007)

Ernst, M. ) (2007). Learning to integrate arbitrary signals from vision and touch.Journal of Vision

work page 2007
[7]

Hsu, J., Mao, J., & Wu, J. (2023). Ns3d: Neuro-symbolic grounding of 3d objects and relations.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2614–2623

work page 2023
[8]

Jacobs, R. A. (2002). What determines visual cue reliability? Trends in cognitive sciences,6(8), 345–350. K¨ording, K. P., Beierholm, U., Ma, W. J., Quartz, S., Tenen- baum, J. B., & Shams, L. (2007). Causal inference in mul- tisensory perception.PLOS ONE,2(9), 1–10

work page 2002

Show all 23 references

[9]

M., Ullman, T

Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017). Building machines that learn and think like people.Behavioral and Brain Sciences,40

2017
[10]

Liu, Z., Li, X., Luo, P., Loy, C.-C., & Tang, X. (2015). Se- mantic image segmentation via deep parsing network.Pro- ceedings of the IEEE international conference on computer vision, 1377–1385

2015
[11]

Nam, H., Ha, J.-W., & Kim, J. (2017). Dual attention net- works for multimodal reasoning and matching.Proceed- ings of the IEEE conference on computer vision and pattern recognition, 299–307

2017
[12]

(1954).The construction of reality in the child

Piaget, J. (1954).The construction of reality in the child. Routledge

1954
[13]

Schertz, J., & Clare, E. J. (2020). Phonetic cue weighting in perception and production.Wiley Interdisciplinary Re- views: Cognitive Science,11(2), e1521

2020
[14]

Vinyals, O

Glaese, A., Chen, J., Pitler, E., Lillicrap, T., Lazaridou, A., . . . Vinyals, O. (2024). Gemini: A family of highly capable multimodal models

2024
[15]

Trommershauser, J., Kording, K., & Landy, M. S. (2011). Sensory cue integration. Oxford University Press

2011
[16]

Wang, Y ., Chen, W., Han, X., Lin, X., Zhao, H., Liu, Y ., Zhai, B., Yuan, J., You, Q., & Yang, H. (2024). Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning.arXiv preprint arXiv:2401.06805

2024 arXiv
[17]

Wason, P. C. (1968). Reasoning about a rule.The Quarterly Journal of Experimental Psychology

1968
[18]

G., & Sch¨on, T

Wills, A. G., & Sch¨on, T. B. (2023). Sequential monte carlo: A unified review.Annual Review of Control, Robotics, and Autonomous Systems,6(1), 159–182

2023
[19]

K., Goodman, N

Wong, L., Grand, G., Lew, A. K., Goodman, N. D., Mans- inghka, V . K., Andreas, J., & Tenenbaum, J. B. (2023). From word models to world models: Translating from natu- ral language to the probabilistic language of thought.arXiv preprint arXiv:2306.12672

2023 arXiv
[20]

H., & Tenenbaum, J

Yildirim, I., Siegel, M. H., & Tenenbaum, J. B. (2016). Per- ceiving fully occluded objects via physical simulation.Pro- ceedings of the 38th annual conference of the cognitive sci- ence society

2016
[21]

M., Wei, M., Zhang, C

Ying, L., Collins, K. M., Wei, M., Zhang, C. E., Zhi-Xuan, T., Weller, A., Tenenbaum, J. B., & Wong, L. (2023). The Neuro-Symbolic Inverse Planning Engine (NIPE): Model- ing probabilistic social inferences from linguistic inputs. arXiv preprint arXiv:2306.14325

2023 arXiv
[22]

L., & Tenenbaum, J

Weller, A., Shu, T., Griffiths, T. L., & Tenenbaum, J. B. (2025). On benchmarking human-like intelligence in ma- chines.arXiv preprint arXiv:2502.20502

2025 arXiv
[23]

Ying, L., Zhi-Xuan, T., Wong, L., Mansinghka, V ., & Tenen- baum, J. B. (2025). Understanding epistemic language with a language-augmented bayesian theory of mind.Transac- tions of the Association for Computational Linguistics

2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

David, B., Finn, C., Fu, C., Gopalakrishnan, K., Haus- man, K., et al. (2022). Do as I can, not as I say: Grounding language in robotic affordances.arXiv preprint arXiv:2204.01691

arXiv 2022

[2] [2]

Alais, D., & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration.Current biology, 14(3), 257–262

work page 2004

[3] [3]

W., Hamrick, J

Battaglia, P. W., Hamrick, J. B., & Tenenbaum, J. B. (2013). Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences,110(45), 18327–18332

work page 2013

[4] [4]

W., Jacobs, R

Battaglia, P. W., Jacobs, R. A., & Aslin, R. N. (2003). Bayesian integration of visual and auditory signals for spa- tial localization.Josa a,20(7), 1391–1397

work page 2003

[5] [5]

A., & Wang, H

Elizalde, B., Deshmukh, S., Ismail, M. A., & Wang, H. (2022). Clap: Learning audio concepts from natural lan- guage supervision

work page 2022

[6] [6]

) (2007)

Ernst, M. ) (2007). Learning to integrate arbitrary signals from vision and touch.Journal of Vision

work page 2007

[7] [7]

Hsu, J., Mao, J., & Wu, J. (2023). Ns3d: Neuro-symbolic grounding of 3d objects and relations.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2614–2623

work page 2023

[8] [8]

Jacobs, R. A. (2002). What determines visual cue reliability? Trends in cognitive sciences,6(8), 345–350. K¨ording, K. P., Beierholm, U., Ma, W. J., Quartz, S., Tenen- baum, J. B., & Shams, L. (2007). Causal inference in mul- tisensory perception.PLOS ONE,2(9), 1–10

work page 2002