Pith.

Robotics · arXiv:2611.09794 v2 · submitted 6 Nov 2026

Seeing What Execution Needs: Benchmarking Task-Conditioned Actionable Spatial Belief for Manipulation

Yina Jian, Mingkun Xu

The paper says

“When should a robot look at the cup, and when at the portafilter? A view that is informative for one task can be useless for another, even at the same workstation.”

more of the abstract

We introduce Seeing What Execution Needs, a benchmark for task-conditioned actionable spatial belief in simulated tabletop manipulation. Its aligned three-domain track contains 1,080 frozen cases (360 each for coffee workstation, drawer/cabinet, and microwave/appliance). The primary coffee protocols – closed-loop actionability and same-object functional out-of-distribution generalization – produce over 115,000 evaluator episodes across evaluated methods, budgets, and inference repeats. Each case pairs candidate RGB views with a manipulation command, shared scripted skills, and hidden postconditions, directly measuring whether selected observations support execution. A metadata-assisted diagnostic reference pipeline maintains spatial-functional beliefs and ranks views by missing evidence. At primary budget settings, it reaches 98.8% closed-loop and 98.9% functional-OOD success; removing functional fields or command binding reduces functional-OOD success to 0.0% and 19.7%, respectively. Under a separate full-preview contract, seven vision-language models reach 26-43%. Replacing coverage metadata with frozen RGB-derived scores retains 98.5%/98.9% success; shuffling these scores reduces success to 15%/11%, supporting dependence on image content. Separately, the aligned three-domain evaluation at H=3 supports portability of the benchmark interface under frozen domain-specific schemas. These results suggest that spatial belief for manipulation should be evaluated by whether acquired evidence meets the current command's execution requirements.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 9 Nov 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A carefully-built command-conditioned view-selection benchmark with unusually honest controls; the reference pipeline mirrors its own rubric, which the authors disclose but never quantify, yet the narrow central claim holds. This line is wrong

3 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at §5, Table 2 (RGB-derived coverage)

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The near-ceiling VBM-VISION result does not resolve the rubric-circularity concern it is presented against. The per-predicate coverage head is trained on predicate-visibility labels drawn from the same functional taxonomy that defines the command templates R(τ) in Section 3 and that the hidden postconditions check, as the Fig. 5 trace shows for the grind case.

more of objection 1

The shuffle control (14.9%/11.4%) proves that the predicted coverage rows carry image content, but it does not bound the contribution of the label prior, and the Discussion acknowledges only that 'a fully unsupervised variant remains future work' without quantifying that contribution. To support the interpretation that the 98.5% result and the 55.9-point VLM gap measure command-conditioned evidence acquisition rather than rubric possession, please add at least one quantitative control: (i) a VLM condition in which the functional-field rubric is supplied in-context, (ii) a VBM-VISION variant trained on a permuted predicate taxonomy, or (iii) an explicit decomposition of final success into the coverage-head contribution versus the hand-authored command-binding contribution. Without such a control, the abstract's headline gap remains ambiguous between 'cannot acquire the evidence' and 'does not possess the rubric.'

The paper says, at §3, Eq. (1) (View Belief Map)

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The View Belief Map scoring rule is under-specified for the claims built on it. Equation (1) defines J(v;B_t,τ) = (1/|M_t|) Σ g(v,p_i,τ) − λ c(v), but the paper gives neither the ordinal evidence-gain table g(v,p_i,τ) over the benchmark's (view type, predicate) pairs nor the value of λ or the normalization of c(v), describing g only as a 'deterministic ordinal score.'

more of objection 2

Since the auditable reference pipeline is a central deliverable and Tables 1 and 15 report ablation contrasts built on this rule, please tabulate g and λ, report a sensitivity sweep over λ and over coarsenings of the ordinal g scale, and confirm that the 98.8%/98.9% results and the collapse patterns in Table 15 persist across the sweep.

The paper says, at §3 Task interface; §5 RGB-derived coverage

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The paper should state explicitly whether the predicate-visibility labels used to train the VBM-VISION head are part of the public training data available to all benchmark users. The task-interface paragraph says the agent never sees 'hidden predicates, evaluator labels,' but that describes the reference pipeline's operation, not the training access of future benchmark entrants; if the labels are public, a …

more of objection 3

… method can approach the near-ceiling results simply by learning the label taxonomy, and the structure of the hidden postconditions is recoverable from public labels plus command templates. Please clarify what exactly is released (predicate-visibility labels, view metadata, command templates) and, if the labels are public, add a baseline that uses only RGB and commands without taxonomic labels so that the benchmark's headroom for label-free selectors is documented.

4 minor objections

The paper says, at Abstract and §1

Pith says: minor objection 1 of 4

The candidate-view counts are inconsistent across domains: the abstract and Section 1 say every case pairs with twelve frozen candidate views and report 860 cases with 6,000 candidate-view records, but 860×12 = 10,320, while supplement Table 10 reports 1,800 rendered views per 180-case held-out domain (10 views per case), giving 9,600 total; please state the per-domain candidate-view counts explicitly.

The paper says, at Abstract; §5 Foundation-model stress test

Pith says: minor objection 2 of 4

The 26.0-42.9% VLM range is presented in the abstract without the full-preview and non-controlled-comparison caveat that the paper correctly states in the Discussion and supplement F.1; adding a clause such as 'under a full-preview ranking contract that is not a controlled comparison with the sequential reference' would prevent the headline from being read as an equal-contract comparison.

show 2 more

The paper says, at Supplement Table 16

Pith says: minor objection 3 of 4

The C.1 ablation full-method value (97.60%) differs from the primary-run 98.8% in Table 1; the surrounding text notes this, but the table caption should state it as well, since the two tables are likely to be compared directly by readers.

The paper says, at §3, ledger definition

Pith says: minor objection 4 of 4

The label 'stale' in the definition of M_t is not operationalized: Section 3 says κ_i is raised only by actions that may invalidate the corresponding predicate type, but it does not list which actions raise κ_i for which predicate types, and a one-line specification of that mapping would make the ledger update rule fully reproducible.

What would settle it

From the released frozen episode logs, compute, within each (task, protocol, budget) cell, P(success | all target functional predicates covered by the selected views) for the reference and VBM-VISION rows. If this probability is ≈1.0 in every cell, the hidden postcondition is entailed by rubric coverage and the near-ceiling results are rubric-coverage results whose taxonomy is injected through labels and templates: the circularity concern lands and the primary numbers need re-scoping. If it is materially below 1.0, or a substantial share of successes occur without full target coverage, the evaluator carries physical information beyond the author-defined rubric, largely clearing the concern. The frozen logs already contain selected-view IDs, target-field hits, and postcondition outcomes, so the check needs no new simulation.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 9 Nov 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 2 min 36 s · 30,798 tokens read, 20,395 written
  3. Editor wrote the verdict above 43 s · 31,810 tokens read, 5,821 written
  4. Referee major revision: 3 major objections, 4 minor 1 min 59 s · 31,168 tokens read, 15,333 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Two things to know. First, the reframing is useful: score view selection by whether it exposes the evidence the current command needs, not by scene-level completeness. That is a real complement to Theory of Space, ESI-Bench, and BEHAVIOR-Spatial. Second, the experimental hygiene is better than most benchmark papers I see, and the main caveat is one the authors state but never resolve: the reference pipeline's design mirrors the benchmark's own rubric, so the headline VLM gap partly measures rubric possession, not just evidence acquisition.

What is actually new: frozen cases with hidden postconditions, budget sweeps, same-object functional-OOD splits (same cup, different commands need different evidence), an H=1 stress split, and a genuinely strict-clean VLM admission gate. The ablations carry real weight: removing functional fields collapses C.2 to 0.0%, removing command binding to 19.7%. The shuffle control (15%/11%) shows the positive result depends on image content. The authors report the H=1 collapse to 19.0% in C.1 and 58.6% in C.2 instead of hiding it, and they say plainly that the primary splits are saturated, that the VLM comparison is not controlled, and that Drawer/Microwave are sanity checks, not transfer evidence. The provenance package (manifests, raw records, audit chain) is what a benchmark submission should look like.

The soft spot, in proportion: command templates enumerate the exact functional fields the postconditions check, and the view metadata maps candidate views to those same fields. The 98.8% reference result is partly a consistency check of the authors' own taxonomy, and the 26-43% VLM gap conflates having that taxonomy with acquiring evidence. VBM-VISION removes the metadata at inference but keeps predicate-visibility labels from the same taxonomy at training time; the stress-test note is right that this is acknowledged (b194) but unquantified. The decisive referee ask is a permuted-predicate control and a VLM run that is given the predicate vocabulary.

The narrow central claim survives. The no-image control fails at 0.0%, the category shortcut fails, and H=1 shows real difficulty even with full rubric knowledge. So the benchmark measures command-conditioned evidence selection; it just cannot support the widest framing by itself.

This paper is for people building or using VLM/VLA manipulation evaluations. It deserves a serious referee; I would accept and request the label-prior quantification. Reading group yes - it will start a good argument about how much of their own taxonomy a benchmark is allowed to encode.

The strongest objection

The paper's strongest defense against the metadata-circularity objection is VBM-VISION (Section 5, Table 2): replacing the reference pipeline's simulator-derived coverage metadata with frozen CLIP-B/32 features plus a per-predicate linear head preserves 98.5% (C.1) and 98.9% (C.2). But the head is trained on predicate-visibility labels authored from the same functional taxonomy that defines the command templates R(τ) and, per the logged trace in Fig. 5, the same fields scored by the hidden postconditions (3/3 passed against three target functional fields). The paper acknowledges this dependency (b194: 'a fully unsupervised variant remains future work'), yet nothing quantifies how much of the near-ceiling result is carried by the label prior rather than by image content. The shuffle control (15%/11%) shows the scores carry image information; it does not show that near-ceiling success reflects command-conditioned evidence acquisition rather than reproduction of the authors' rubric, because the rubric vocabulary is injected at training time and the command binding is hand-authored to match the evaluator's predicate decomposition. Consequently the headline gap (98.8% reference vs. 26.0-42.9% VLMs) conflates possessing the hand-authored functional taxonomy with acquiring the evidence a command needs. The narrow central claim survives: the ablations show command binding and functional fields are decisive (b51, Tab. 15). But the benchmark's advertised value as a general measuring instrument for learned view selectors rests on the unquantified label-prior contribution, and the decisive empirical question is whether the hidden postcondition is entailed by coverage of the target predicate set in the selected views.

The reader's weakest assumption and this stress-test converge on the same mechanism: the reference pipeline and its RGB counterpart inherit a hand-authored functional taxonomy whose vocabulary matches the evaluator's predicate decomposition. I place the load-bearing edge on VBM-VISION because it is the paper's principal rebuttal to the metadata-objection: it removes the view-metadata prior at inference, but the coverage head is trained on predicate-visibility labels from that same taxonomy and the command binding remains hand-authored (supplement F.1; b194 concedes the fully unsupervised variant is future work). The shuffle control establishes that the scores carry image information; it does not establish that near-ceiling success reflects command-conditioned evidence acquisition rather than faithful prediction of hand-authored labels, because the labels, templates, and postcondition fields share one vocabulary (Fig. 5's trace aligns 3/3 passed postconditions with the three target functional fields). Independent support is genuine: component-removal ablations (C.2 w/o functional fields → 0.0%; w/o command binding → 19.7%, Tab. 15 and b51), the H=1 stress split (19.0%/58.6%, b52), the paired scene-group bootstrap (Tab. 6), the shuffle control, and unusually explicit limitation disclosures (b149, b194, b198, supplement L) all hold up. These keep the narrow central claim intact: under a fixed scripted-skill interface, the ablations establish the causal role of task-conditioned spatial-functional selection inside the benchmark's own rubric. What remains unsettled is the benchmark's external validity as a measuring instrument: whether the hidden postconditions carry information beyond rubric coverage. The concrete test I propose settles exactly that, using data the authors already release. If entailment holds, the primary numbers and the VLM gap should be re-read as rubric-coverage results, and the benchmark's promotion as a general evaluation of manipulation spatial belief should carry the reader's conditions (unsupervised RGB variant, controlled VLM contract, cross-domain or real-robot validation). If the evaluator carries independent physical information, the circularity concern is largely resolved. Either way the reader's CONDITIONAL verdict stands; no adjustment is needed.

The referee's report

The contribution is timely and well-scoped: it operationalizes 'actionable spatial belief' with a controlled interface that isolates evidence acquisition from controller quality, and the reference pipeline is unusually auditable. The paper's credibility is strengthened by concrete practices: a paired scene-group bootstrap over 500 frozen case clusters, a strict-clean admission gate for all seven VLM rows with raw responses retained, a shuffle control on the RGB-derived coverage scores, logged rollout traces, SHA-256 manifests, and explicit limitation statements covering the single workstation, scripted skills, and the non-controlled VLM comparison. If the central claim is accepted, the benchmark provides a valuable complement to scene-level spatial-reasoning suites and a reproducible yardstick for learned view selectors.

Pith's reading of the paper (a machine's summary, not the authors')

The paper sets out to establish that spatial belief for manipulation should be judged by actionability: whether the observations a robot acquires make the current command executable, not how completely it describes the scene. It introduces a frozen RGB benchmark of 860 simulated tabletop cases — a five-task coffee workstation plus drawer/cabinet and microwave/appliance interface domains — in which each case pairs a manipulation command with twelve candidate views, shared scripted skills, and hidden simulator-state postconditions. A diagnostic reference pipeline that keeps a command-conditioned spatial-functional belief ledger and ranks candidate views by expected gain over missing evidence reaches 98.8% closed-loop success at a five-view budget and 98.9% same-object functional-OOD success at three views, while seven vision-language models that preview all candidates and return three ranked views reach only 26–43%. If the paper is right, a robot can look at the right scene and still acquire the wrong evidence, and benchmarks that score scene-level completeness will miss exactly what manipulation requires.

What carries the argument. The load-bearing object is the Spatial–Functional Belief Ledger, a running record of entries $(p_i, \hat{y}_i, q_i, E_i, \kappa_i, \rho_i)$ — predicate name, estimated value, confidence, visual provenance, staleness flag, task-relevance weight — bound to the current command through hand-authored command templates $R(\tau)$. The mechanism that carries the argument is the View Belief Map: a scoring rule that ranks candidate views by expected gain over the missing, uncertain, or stale target-relevant predicates in the ledger, producing a discrete command-conditioned score over the candidate cameras rather than a dense scene model. Optional Action-Induced Belief Revision compares predicted with observed postconditions after execution and re-observes only the contradicted predicate at a charged cost. The paper positions this pipeline as an auditable metadata-assisted anchor, not a learned perception model, and the ablation matrix is what demonstrates that functional fields, command binding, and target conditioning each carry the functional-OOD result individually.

Core claim. The paper's central claim is a dissociation: scene knowledge and actionable evidence come apart. A belief state organized around command-relevant functional predicates — part visibility, relative pose, support, containment, control state — plus a rule that scores each candidate view by its expected gain over the missing, uncertain, or stale predicates in that state, is what converts spatial evidence into task success. The reference implementation maintains this in a Spatial–Functional Belief Ledger and scores views with a View Belief Map, $J(v;B_t,\tau) = \frac{1}{|M_t|}\sum_{p_i \in M_t} g(v,p_i,\tau) - \lambda c(v)$, where $M_t$ is the missing-predicate set, $g$ an ordinal evidence-gain estimate, and $c(v)$ a normalized view cost. On the coffee domain this reaches 98.8% closed-loop success at budget $H=5$ and 98.9% same-object functional-OOD success at $H=3$, with target-cue observation tracking success while wrong-command cue observation does not. The ablation is the load-bearing evidence: removing functional belief fields collapses functional-OOD success to 0.0%, removing command binding to 19.7%, removing target conditioning to 3.5%, and a category shortcut reaches 3.7%. Replacing simulator-derived coverage metadata with frozen RGB-derived per-predicate visibility scores preserves 98.5%/98.9% success, and shuffling those scores collapses success to 14.9%/11.4%, which the paper treats as confirmation that the result depends on image content.

The premise it rests on. The near-ceiling scores rest on the benchmark designers having already written down, for every command and every candidate view, which functional parts matter — command templates and coverage metadata that mirror the hidden postcondition structure the evaluator rewards, with the RGB-trained coverage head itself supervised by those same functional-role labels.

If the paper is right (Pith's inferences, not the authors')

Spatial-belief benchmarks for manipulation should score whether the acquired evidence makes the current command executable; the paper's cue diagnostics show target-cue observation tracks success while wrong-command cues do not, so scene-level completeness is the wrong metric.

Same-object functional-OOD is where the claim is strongest: category shortcuts (3.7%), spatial-only views (0.0%), and command-agnostic affordance viewing (19.7%) all fail under the hidden evaluator while the command-bound ledger reaches 98.9%.

Generic vision-language ranking under a full-preview contract leaves a systematic gap: seven strict-clean models cluster in a 26–43% band, and even the strongest sits 55.9 points below the reference at the primary setting.

The benchmark is not solved under every observation regime: with a hard one-view initial budget the reference drops to 19.0% closed-loop and 58.6% functional-OOD success, explicitly leaving headroom for learned selectors.

Future VLM/VLA manipulation evaluation should test whether a model acquires the evidence the next action requires rather than only what it knows about the scene — the paper's stated forward-looking conclusion.

The headline gap between the reference selector and the best vision-language model conflates contract with capability: sequential acquisition backed by a structured ledger is compared against one-shot full-preview ranking, so a matched-contract test — a VLM choosing views sequentially with feedback, or the reference forced to rank blindly in one pass — would attribute, and probably shrink, the 55.

The benchmark's honest open problem may be the H=1 stress split (19.0%/58.6%) and the cross-domain sanity checks (about 67%), which sit far from the near-ceiling primary numbers; the reference's design-prior advantage is most visible exactly where its hand-authored structure is most needed.

The paper's diagnostic QA numbers (46–49% across methods while postcondition success ranges from 0–99%) imply that the answerability of selected images is nearly uninformative about actionability — a caution that scene-level spatial QA scores should not be read as manipulation-readiness.

A natural extension the benchmark enables: hold the workstation fixed and test whether a selector can discover a new functional chain — a command family whose required predicates appear in no template and no training label — which would separate binding known functional structure from discovering new structure.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

When should a robot look at the cup, and when at the portafilter? A view that is informative for one task can be useless for another, even at the same workstation. We introduce Seeing What Execution Needs, a benchmark for task-conditioned actionable spatial belief in simulated tabletop manipulation. Its aligned three-domain track contains 1,080 frozen cases (360 each for coffee workstation, drawer/cabinet, and microwave/appliance). The primary coffee protocols – closed-loop actionability and same-object functional out-of-distribution generalization – produce over 115,000 evaluator episodes across evaluated methods, budgets, and inference repeats. Each case pairs candidate RGB views with a manipulation command, shared scripted skills, and hidden postconditions, directly measuring whether selected observations support execution. A metadata-assisted diagnostic reference pipeline maintains spatial-functional beliefs and ranks views by missing evidence. At primary budget settings, it reaches 98.8% closed-loop and 98.9% functional-OOD success; removing functional fields or command binding reduces functional-OOD success to 0.0% and 19.7%, respectively. Under a separate full-preview contract, seven vision-language models reach 26-43%. Replacing coverage metadata with frozen RGB-derived scores retains 98.5%/98.9% success; shuffling these scores reduces success to 15%/11%, supporting dependence on image content. Separately, the aligned three-domain evaluation at H=3 supports portability of the benchmark interface under frozen domain-specific schemas. These results suggest that spatial belief for manipulation should be evaluated by whether acquired evidence meets the current command's execution requirements.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-09

The authors' main claim, as the classifier picked it out 1.00

“These gaps suggest that spatial belief for manipulation should be evaluated not by scene-level completeness, but by whether it acquires the evidence the current command requires.”

How the main claim is stated: hedged possibility 0.98

What the abstract itself states, as probabilities: claims first 0.12 · states limitation 0.30 · human participants 0.03 · controlled comparison 0.87 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.94 · mentions code or data availability 0.24

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
When should a robot look at the cup, and when at the portafilter? background 0.87 0.04 0.02 unclear 1.00
Spatial-reasoning benchmarks evaluate whether a model can describe a scene, but manipulation needs something more specific: the ability to acquire the particular evidence that makes the current command executable. gap 0.72 0.08 0.18 asserted 1.00
A view that is informative for one task can be useless for another, even at the same workstation. background 0.77 0.12 0.07 asserted 0.99
We introduce Seeing What Matters, a benchmark for task-conditioned actionable spatial belief in simulated tabletop manipulation. contribution 1.00 0.94 0.07 asserted 0.99
Across three domains--coffee workstation, drawer/cabinet, and microwave/appliance--the benchmark provides 860 frozen evaluation cases; expanding across methods, budgets, and inference repeats yields more than 115,000 evaluator episodes under two protocols: closed-loop actionability and same-object functional out-of-distribution generalization. method 0.68 0.92 0.05 asserted 1.00
Each case pairs a set of candidate RGB views with a manipulation command, shared scripted skills, and hidden postconditions, so the benchmark directly measures whether a view-selection policy acquires the observations its command actually needs. method 0.96 0.89 0.08 asserted 1.00
We also provide a diagnostic reference pipeline that maintains a spatial-functional belief state and ranks views by missing evidence. method 0.57 0.92 0.04 asserted 0.99
At primary budget settings, this pipeline reaches 98.8% closed-loop and 98.9% functional-OOD success; ablating the functional or command-binding components collapses performance to near zero. result 1.00 0.95 0.11 asserted 1.00
Seven vision-language models, evaluated on the same cases, reach only 26-43%. result 1.00 0.92 0.07 asserted 1.00
Replacing the reference pipeline's coverage metadata with frozen RGB-derived scores preserves 98.5%/98.9% success; shuffling the visual scores collapses it to 15%/11%, confirming that the result depends on image content. result 0.98 0.96 0.07 asserted 0.99
These gaps suggest that spatial belief for manipulation should be evaluated not by scene-level completeness, but by whether it acquires the evidence the current command requires. interpretation 1.00 0.54 0.87 hedged possibility 0.98

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from simulation or computation 0.72

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-09: 1 request, 8,384 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (18)
Figure 1 of the paper
Figure 1. Seven-model evidence diagnostics. Left: success versus target-cue rate. Right: success versus wrong-command cue rate. Seven strict-clean VLMs each have 1,500 evaluation records, pre￾view twelve RGB candidates, and select exactly three views. Dots show point estimates; icons label models. Cue rates are separate evaluator diagnostics, not complementary fractions. 1. Introduction An embodied agent can look at the right scene and still acquire the wrong evidence. Consider a coffee worksta￾tion: tamping requires observing the portafilter basket, puck surface, and tamper alignment; cup…
Figure 2 of the paper
Figure 2. From spatial belief to actionable evidence. Theory-of-Space-style evaluation asks whether an agent can construct and revise spatial beliefs through exploration. Seeing What Matters keeps the active-belief motivation but scores whether command-conditioned views expose the evidence needed by a shared scripted skill, with hidden postconditions used only for evaluation. Schematic modules are illustrative. parts, action regions, and manipulation trajectories [4, 19, 25, 29, 48], while active-vision systems learn where to look before acting [1, 2, 18, 21, 32, 41, 44]. These directions…
Figure 3 of the paper
Figure 3. Task-conditioned evidence acquisition. The cof￾fee workstation supplies the quantitative five-task benchmark; Drawer/Cabinet and Microwave/Appliance are interface illustra￾tions whose small quantitative sanity checks appear in the sup￾plement. (a) The command and ledger identify missing evidence. (b) The transparent View Belief Map ranks candidate views. (c) Ac￾quired observations update the ledger before a shared scripted skill executes. (d) Post-action observations support optional local revi￾sion; hidden simulator state is reserved for evaluation. This hybrid schematic combines…
Figure 5 of the paper
Figure 5. Logged rollout trace for command-conditioned view selection. This panel is generated from a real frozen functional￾OOD trace. It shows the candidate-view scores, selected view, observed fields, target functional predicates, and hidden postcon￾dition outcome for a successful one-view GRIND episode with a TAMP distractor command. The trace illustrates the audited deci￾sion that the benchmark scores: the policy succeeds by selecting the view that exposes the command-relevant functional chain before execution. This is one representative successful trace; hard H=1 aggregate success is…
Figure 6 of the paper
Figure 6. Primary five-task visual domain. Raw MuJoCo/Panda RGB exemplars from the coffee workstation show that tamping, insertion, placement, serving, and grinding require different command-relevant functional evidence even when the object family is shared. Raw MuJoCo candidate-view evidence: held-out domains Drawer/Cabinet front wide cabinet view with open drawer handle close-up: drawer pull handle interior top-down: interior support surface target close-up: target object in drawer Microwave/Appliance front handle interior control Small cross-domain sanity checks use the same logged RGB…
Figure 7 of the paper
Figure 7. Raw MuJoCo candidate-view examples for held-out domains. Drawer/Cabinet and Microwave/Appliance use the same logged RGB candidate-view interface and hidden simulator-state postcondition evaluator as the coffee benchmark. E. Paired Scene-Group Bootstrap
Figure 8 of the paper
Figure 8. Seven-model strict-clean comparison. (a) Overall postcondition success with archived normal-approximation 95% half-width. (b) Per-task success. (c) Target-cue, wrong-cue, and selectivity diagnostics. Generated only from the archived strict-clean CSVs; it does not equate the VLM full-preview contract with sequential acquisition. Model Success ↑ Target ↑ Wrong ↓ Select. ↑ Coll. ↓ Post-fail ↓ n Gemma 3 27B 32.5 ± 2.4 63.8 28.1 35.6 3.7 63.7 1,500 Grok 4.3 42.9 ± 2.5 85.2 28.2 57.0 2.6 54.5 1,500 Grok 4.6 42.5 ± 2.5 80.2 27.8 52.4 2.4 55.1 1,500 Llama 4 Maverick 28.3 ± 2.3 66.3 32.3…
Figure 9 of the paper
Figure 9. Task success versus initial observation budget. Lines connect only the observed budgets H ∈ {1, 3, 5, 8}. Recovery methods may use extra charged recovery observations after the initial budget, which are analyzed separately in the supplementary charged-view curves. generated figures and tables, no-functional-fields success is 0.0%; the 16.7 value refers to target-cue observation. The source package contains regenerated CSVs and verification records under data/, MuJoCo run and validation scripts under scripts/, and review records including REPRODUCIBILITY.md, DATA_CARD.md, ASSET_ATTR
Figure 10 of the paper
Figure 10. Hard one-view budget stress test. The figure is gener￾ated from the frozen episode logs by filtering to H = 1. It exposes residual difficulty that is obscured by the primary H = 5 C.1 and H = 3 C.2 settings. Missing protocol/method combinations are left unconnected rather than imputed. Uncertainty for reported rows is provided in
Figure 11 of the paper
Figure 11. Real logged rollout trace. Candidate-view scores, selected view, observed fields, and postcondition outcome are read from the frozen trace record rather than drawn as a storyboard. This is one representative successful trace; the hard H = 1 aggregate success rates are reported in
Figure 12 of the paper
Figure 12. Average charged views versus initial observation budget. Left: C.1 closed-loop actionability. Right: C.2 functional-OOD. Recovery observations are charged in addition to the initial selection budget; lines connect only observed budgets H ∈ {1, 3, 5, 8}. Protocol Variant Tamp Insert Place Serve Grind Average C.1 Full 97.0 98.3 96.7 98.0 98.0 97.6 C.1 w/o AIBR 96.7 98.7 98.3 98.3 98.7 98.1 C.1 w/o belief memory 0.0 39.3 2.0 97.3 98.0 47.3 C.1 w/o functional fields 99.3 99.3 98.7 98.7 99.0 99.0 C.1 w/o target conditioning 0.0 0.0 0.0 0.0 0.0 0.0 C.2 Full 99.0 99.0 99.0 98.7 99.0 98.9…
Figure 14 of the paper
Figure 14. Recovery stress test for postcondition mismatch. Optional Action-Induced Belief Revision repairs stale belief fields locally and restores 98.7% of mismatch episodes in the constructed mismatch setting. The right panel reports total charged views including initial observations, not only extra recovery views. promoted
Figure 15 of the paper
Figure 15. Closed-loop task success by task at H = 5 and functional-OOD task success by task at H = 3. (a,b) Task-level matrices show the five task families and the corresponding aggregate. Same-object command shifts expose category shortcuts. 0 2 4 6 8 10 Random Fixed scan Coverage History Ledger VBM VBM+AIBR Priv. diag. 6.2 8.1 4.4 2.3 3.1 2.1 1.2 1.9 (a) C.1 — Collision rate (%) 0 20 40 60 80 100 Random Fixed scan Coverage History Ledger VBM VBM+AIBR Priv. diag. 82.7 91.9 95.6 44.7 84.5 (b) C.1 — Postcondition failure (%) 0 2 4 6 8 10 Random Fixed scan Coverage History Ledger VBM VBM+AIBR…
Figure 16 of the paper
Figure 16. Closed-loop failure metrics at H = 5. Collision and postcondition failure are displayed separately. Functional-OOD failure metrics at H = 3. Failure rates expose policies that observe plausible but command-irrelevant evidence. All rates use 1,500 episodes per method/protocol as denominator; the two failure metrics are nonexclusive. Protocol-separation rules. The following records are never silently merged: (i) the 180-case/domain Drawer/Microwave check and the separate 360-case/domain learned-selector branch; (ii) the C.1 ablation-run full-method value 97.60% and the 98.8% primary…
Figure 17 of the paper
Figure 17. Recovery after mismatch by task. AIBR is evaluated as a repair diagnostic rather than the main source of primary-split gains. Denominators are 150 episodes per task/method and 750 per displayed method average. 40 45 50 55 QA accuracy (%) Random Fixed scan Coverage History Ledger 46.3% 47.8% 46.0% 48.6% 48.2% Diagnostic QA accuracy (H=5)
Figure 18 of the paper
Figure 18. VLM diagnostic QA accuracy at H = 5. QA behavior is reported as a diagnostic of whether selected images contain an￾swerable spatial-functional information; it is not treated as the main manipulation result. Each point reports correct_count/n with n = 1,500; the normal-approximation 95% binomial half-width is about 2.5 points. No random-answer chance level is defined; Random denotes random view selection. Benchmark component Fixed and auditable element Task-conditioned action￾able belief Evidence is judged by whether it makes the current command executable, not by generic scene QA…

Showing 16 of the paper's 18 figures. Figures are the authors', reproduced from the paper.

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment