Pith.

Computer Vision and Pattern Recognition · arXiv:2610.21783 v2 · submitted 15 Oct 2026

Learning to Think and Explore in 3D World

Yijiang Li, Bingyang Wang, Nuno Vasconcelos

The paper says

“We study how vision-language models (VLMs) can learn to think and actively explore in 3D space.”

more of the abstract

Instead of providing multi-view images, videos, or 3D representations to the model, we preserve the native visual encoding of VLMs, i.e., a single image, one at a time, just like humans, and treat the 3D representation only as an environment with which the model can interact through actions (e.g., move left) and observations (rendered images). Each rollout begins from an initial first-person view and a visual question. The model can then iteratively reason, navigate, and reorient its viewpoint before committing to a final answer. Following an initial trajectory-supervised finetuning, we apply reinforcement learning (RL) with rewards driven primarily by final-answer correctness. Crucially, RL provides no direct supervision over which viewpoints the model should visit or which exploration actions it should take. Instead, the model must learn an effective exploration policy solely from the extent to which its interactions improve performance. Across six 3D benchmarks, our model ACT3D achieves an average accuracy of 53.2% that improves its base Qwen model by 22.0% and surpasses proprietary Gemini 3.6 Flash. ACT3D also outperforms existing open-source 3D methods while using 50-70% fewer input frames, and generalizes to unseen domains with an average improvement of 5.9% over its supervised initialization. Further analysis reveals adaptive, question-conditioned exploration. Performance gains over the base model increase from 0.6% on questions answerable from the initial view to 16.3% on those requiring extensive exploration. ACT3D searches for occluded targets, avoids unnecessary exploration, and acquires complementary viewpoints for spatial comparisons. These findings demonstrate that outcome-based RL enables VLMs not only to reason over visual evidence, but also to learn when and how to acquire it.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 16 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A solid new formulation for learning to explore 3D scenes with RL, but every headline number comes from an uncalibrated LLM judge that also supplied the training reward. This line is wrong

4 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at Sec. 4.1; App. C.3; Eq. (8); Table 1

The review did not quote this passage.

Pith says: Objection 1 of 4 · major

The headline accuracies (53.2% average, +22.0% over base, +1.0% over Gemini, +4.6% RL-over-SFT) are all verdicts of the Qwen3-30B-A3B-Instruct judge, and the training reward in Eq. (8) appears to use the same model for R_judge, since App. C.3 describes the training signal as coming from a 'local judge' with temperature 0.1 while Sec. 4.1 names Qwen3-30B-A3B-Instruct for evaluation.

more of objection 1

The paper reports no human calibration, no agreement statistic, and no official-metric counterpart for Table 1, so judge leniency or a format preference for terse Qwen-style outputs cannot be separated from genuine accuracy gains; the 1.0-point margin over Gemini lies exactly in the range such a bias could explain. Please state explicitly which judge model is used in training versus evaluation, report human-judge agreement on a stratified sample together with per-model judge error rates, and provide official metrics (EM/CIDEr/ROUGE) for at least the key rows of Table 1, since Appendix B shows the gap between judged accuracy and EM can be large (Ross3D: 50.7 versus 30.7).

The paper says, at Sec. 3.3, Eq. (9)

The review did not quote this passage.

Pith says: Objection 2 of 4 · major

The claim that RL provides no direct supervision over which viewpoints the model should visit is not supported as stated. R_gnd uses the annotated 3D location of the answer object, and c(tau) pays the entire interaction budget N_max as soon as that object is visible in some rendered view, so the reward directly supervises an intermediate property of the selected viewpoints, not only final-answer correctness.

more of objection 2

At the stated weights (lambda_int = 0.05, N_max = 12) this term can contribute 0.60, larger than lambda_judge = 0.30, so it is a substantial part of the training signal. The authors acknowledge that the grounding term 'only verifies whether the answer object was observed,' but that verification is itself a form of evidence-level supervision. Please report how often R_gnd fires during training and add an ablation without the grounding term; without this, the paper's central conceptual claim that exploration is learned purely from task outcome is not established.

The paper says, at Table 1; App. B; Eq. (6)

The review did not quote this passage.

Pith says: Objection 3 of 4 · major

The scoring protocol for Table 1 should be pinned down per baseline. Section 4.1 says the Qwen judge is used throughout evaluation, but Eq. (6) defines accuracy as exact equality with y*, and the Ross3D ScanQA entry (50.7) is the same number as the reproduced ROUGE-L in Table 3 while the official ScanQA EM is 30.7.

more of objection 3

If Table 1 mixes judge-based semantic accuracy for Act3D with official metric scores for baselines, the comparisons are not valid; if all rows are judge-based, the paper should say so explicitly and should still provide official-metric counterparts, because the judge's threshold is effectively the quantity being measured. Please also report evaluation set sizes and confidence intervals for the headline deltas, at least for the 1.0-point lead over Gemini and the 4.6-point RL-over-SFT gain.

The paper says, at Sec. 3.3, Eq. (9)

The review did not quote this passage.

Pith says: Objection 4 of 4 · major

The visibility predicate R_gnd is not operationalized. Determining whether the answer object is 'visible in some rendered view' requires a decision about frustum culling, occlusion by scene geometry, object extent (center, bounding box, or mask), and the rendering/depth procedure; the paper also does not state how the annotated 3D location is obtained for each training source (ScanQA, MP3D, curated ScanNet and ARKitScenes).

more of objection 4

Since the interaction reward, and hence the learned exploration policy through Eq. (11), depends directly on this predicate, please define it precisely and state its availability across datasets.

4 minor objections

The paper says, at Table 2 vs. Fig. 8

Pith says: minor objection 1 of 4

The ScanQA accuracy for the single-rendered-frame configuration differs between Table 2 (41.0) and Figure 8 (41.4); please reconcile the numbers or state the differing training/evaluation conditions.

The paper says, at Sec. 3.1

Pith says: minor objection 2 of 4

The text refers to 'Appendix 4.6' when comparing action parameterizations; this should be Section 4.6.

show 2 more

The paper says, at Fig. 10

Pith says: minor objection 3 of 4

The caption says 'ScanQA Acc.' but the figure reports all six evaluation benchmarks; please reword the caption to say accuracy on the six benchmarks.

The paper says, at Sec. 4.1

Pith says: minor objection 4 of 4

Please state how many evaluation questions and scenes each benchmark contributes and how initial viewpoints are sampled; this is needed to interpret the one-draw standard error in Figure 4(a) and the absence of error bars in Table 1.

What would settle it

Run a blind human-scoring study: draw ~200 predictions per condition (Qwen3.5-9B base, SFT init, Act3D, Gemini 3.6 Flash) from each of the six benchmarks, have three annotators score each against the ground truth with the same rubric, and compute (i) judge-human agreement (Cohen's kappa) plus false-accept/false-reject rates of the temperature-0 evaluation judge, and (ii) per-condition human-scored accuracies and the human-scored RL-over-SFT delta with confidence intervals. If kappa >= 0.9 and the human-scored RL-over-SFT delta stays at or above ~3 points with non-overlapping intervals, the concern is resolved. If the delta collapses or the judge's false-accept rate exceeds roughly 10%, the headline gains are partly judge artifacts and the paper should re-report under the calibrated metric.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 16 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 1 min 22 s · 22,238 tokens read, 10,190 written
  3. Editor wrote the verdict above 56 s · 23,350 tokens read, 6,083 written
  4. Referee major revision: 4 major objections, 4 minor 1 min 16 s · 22,776 tokens read, 10,680 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

You should read this for the setup, not for the headline numbers. Act3D is a clean, honest entry on making a VLM learn where to look in a 3D scene: single egocentric view, relative motion actions, rendered observations from a mesh, trajectory SFT from GPT-5.4, then GRPO over the whole reason-act-observe rollout. As far as the related work shows, that combination is new; prior active methods either select views without training (Chain-of-View) or hand the model a reconstruction tool (Think3D). The paper also does a lot of transparent work around the core claim: Appendix A formalizes the passive observation ceiling, they reproduce Ross3D's published numbers to validate their harness, and they ablate RL-without-SFT, action parameterization, rendering vs. real-image retrieval, and the per-interaction reward. That is more supporting evidence than most papers at this level ship.

The soft spots are real but not fatal. First, every number in Table 1 is an LLM-judge accuracy, and the same model family (Qwen3-30B-A3B-Instruct) supplies the RL reward in training and the evaluation verdicts at test. There is no human calibration, no kappa, no second judge. For Ross3D on ScanQA, judged accuracy is 50.7 versus the reproduced official EM of 30.7. That ~20-point gap means the judge accepts far more than exact matches, and how many of those acceptances a human would endorse is unmeasured. Judge leniency is not damning by itself: all models are judged the same way, and they added a coherence guardrail and an EM term. But the 1.0-point edge over Gemini 3.6 Flash could plausibly be style bias in a Qwen-family judge favoring Qwen-style outputs, and there are no error bars. This needs human agreement data or an alternative judge before the quantitative claims should be taken at face value.

Second, the outcome-based-only pitch is softer than advertised. Equation 9 pays full interaction credit when the annotated answer object becomes visible, a grounding signal that tells the policy which object matters and rewards acquiring evidence about it. The paper discloses this, and it is not action-level supervision, so the central claim survives in weakened form: what is learned is not exploration from pure final-answer reward, but exploration from outcome reward plus a sparse evidence-visibility bonus. The abstract and intro should say that.

Minor nits: no released code, no error bars, the camera passes through geometry. The main work for the authors is the judge. For a reader, the value is the formulation and the careful ablations, not the 53.2%. I would take it seriously as a submission; a strong referee would require judge calibration and reporting on official metrics. I would cite the setup, not the numbers.

The strongest objection

The load-bearing step is unstated: every number in Table 1 (53.2% average, 22.0% over base, 1.0% over Gemini 3.6 Flash, 4.6% RL-over-SFT) is an LLM-judge accuracy, not a human-verified or benchmark-official metric. Section 4.1 states that Qwen3-30B-A3B-Instruct is used as the answer judge throughout evaluation, and Appendix C.3 shows the training reward uses the same model (temperature 0.1) while evaluation uses temperature 0 with a shorter, different prompt. The reward in Eq. 8 is therefore not verifiable in the sense used to motivate outcome-based RL: its largest correctness component (lambda_judge = 0.3) is a model verdict, not ground truth. Because RL optimizes against exactly the standard that later produces the reported accuracies, a systematic judge bias (leniency toward coherent-sounding wrong answers, or preference for the concise Qwen-style outputs Act3D is trained to emit) inflates both the absolute accuracy and the measured RL contribution. Appendix B calibrates the Ross3D reproduction on official ScanQA metrics (EM 30.7 vs 30.8) but not the judge; the gap between the judged Ross3D accuracy in Table 1 (50.7) and its EM (30.7) shows the judge accepts far more than exact matches, and how many of those extra acceptances human raters would endorse is unmeasured. The 1.0-point margin over Gemini sits within the range a format bias could explain, especially since a Qwen-family judge scores a Qwen-based model against a proprietary model with different generation style. The paper does add a coherence guardrail to the judge prompt and retains an EM term (lambda_EM = 0.2) that partially mitigates gibberish exploitation, but no human agreement study, kappa statistic, alternative-judge cross-check, or error analysis of judge verdicts is reported anywhere in the manuscript. Without such calibration, the central quantitative claim is underdetermined.

The reader's weakest assumption and mine coincide: the semantic-equivalence judge underwrites both the training signal (Eq. 8, Appendix C.3) and every reported number (Table 1), and it is never calibrated against humans. I considered three other candidate concerns and found each weaker. (1) The grounding reward in Eq. 9 does provide object-visibility supervision beyond final-answer correctness, but the paper explicitly discloses it ('the grounding term only verifies whether the answer object was observed') and it threatens the 'no direct supervision' interpretation, not the benchmark numbers; it is properly a reason to soften the interpretation, which the reader already does in the rationale. (2) Missing error bars: Figure 4(a) reports a one-draw SE of about 0.7 points, so the 4.6% RL delta is unlikely to be pure noise, though single-seed results remain a limitation. (3) SQA3D and ScanNet: the mixed-data RL corpus includes generated QAs over ScanNet scenes, and SQA3D is built on ScanNet, so SQA3D is not a clean scene-holdout; this weakens only the out-of-distribution generalization claim, not the 53.2% average, which pools in-domain and out-of-domain benchmarks. The judge concern is strictly more load-bearing because every headline number flows through it, and the train/eval coupling (same judge family scoring the reward and the results) means judge bias inflates the measured effect rather than merely adding noise. The paper's transparency about judge prompts (Appendix C.3) and the Ross3D reproduction (Appendix B) are real supporting evidence, but pipeline reproduction is not judge calibration. My read does not move the reader's verdict: the study is well executed, the ablations (action parameterization, rendering vs retrieval, interaction reward) are informative, and the correct disposition remains CONDITIONAL pending the calibration check; if the proposed test passes, the numbers stand, and if it fails, the headline claims must be recomputed under a human-verified metric.

The referee's report

If the reported numbers survive closer scrutiny, Act3D is a useful and timely demonstration that outcome-driven RL can shape not only reasoning but also viewpoint selection in VLMs. The paper has genuine infrastructure strengths: a clearly specified POMDP formulation with deterministic camera dynamics, trajectory SFT before GRPO, a broad six-benchmark evaluation, detailed prompt templates, an ablation of the interaction reward, and a careful reproduction of the Ross3D baseline on official ScanQA metrics. However, the strength of the central claim is currently limited by two unverified pillars: an uncalibrated LLM judge that supplies both the RL reward and every headline number, and a grounding reward that leaks the ground-truth answer object's location into the policy. Neither issue is fatal in itself, and both are addressable with additional analysis and reporting, but as written they prevent the reader from separating genuine task-level learning from judge bias and reward shaping.

Pith's reading of the paper (a machine's summary, not the authors')

This paper argues that a vision-language model can learn to answer questions about a 3D scene by actively moving through it, rather than being handed a complete 3D representation. The system, Act3D, starts from a single first-person image and a question, then alternates private reasoning with camera actions -- move, turn, look -- until it commits to an answer, treating the scene as an external environment it interacts with. The training signal is deliberately sparse: trajectory demonstrations establish the interaction format, then reinforcement learning rewards only whether the final answer was right, plus a small bonus when the object asked about actually appears in a view; no stage tells the model which viewpoints to visit. On six 3D benchmarks Act3D reaches 53.2% average accuracy, beating its base Qwen3.5-9B model by 22.0 points and the proprietary Gemini 3.6 Flash by 1.0 point, with the reinforcement stage contributing 4.6 points beyond supervised fine-tuning. The claim matters because it suggests task-level outcomes alone can teach a model both what to reason about and when to go look for evidence.

What carries the argument. The central mechanism has two parts. The environment wrapper: a POMDP in which the scene $M$ stays hidden and fixed, the camera state $c_t=(e_t,f_t)$ evolves deterministically under the policy's actions, and each step exposes exactly one rendered view $o_t=\mathrm{Render}(M,c_t)$, so the model must acquire evidence rather than receive it. The action interface is relative and camera-centric -- a primitive (forward, backward, left, right, up, down, four turns) paired with an integer multiplier $n_t\in\{1,\dots,8\}$ -- giving 80 scale-invariant moves plus a stop-and-answer action. The reward is what carries the learning: $$R(\tau)=\lambda_{\mathrm{judge}}R_{\mathrm{judge}}+\lambda_{\mathrm{EM}}R_{\mathrm{EM}}+\lambda_{\mathrm{int}}R_{\mathrm{gnd}}\,c(\tau)-\lambda_{\mathrm{fmt}}R_{\mathrm{fmt}},$$ with the judge term an LLM semantic-equivalence verdict and the interaction term gated so that correctly answering rollouts are paid for evidence (capped at $N_{\max}=12$ and short-circuited once the annotated answer object is observed), not for looking. The appendix argument is the theoretical motor: each additional observation can only shrink the set of scenes consistent with the interaction history, conditioning on more observations cannot increase expected answer uncertainty, and interactive policies dominate passive ones because stopping at step zero is the special case of passive prediction.

Core claim. Act3D reframes 3D question answering as a partially observable decision process: the scene is hidden and external to the model, the policy receives exactly one rendered egocentric image per step along with its camera pose, and it chooses among 80 relative actions -- six translations and four rotations, each with a step multiplier from 1 to 8 -- or a terminal answer. Training has two stages: trajectory-supervised fine-tuning on 5,712 curated demonstrations (an LLM teacher executing reason-act-observe rollouts) establishes the interaction protocol, and then group-relative policy optimization (GRPO) maximizes a reward built almost entirely from outcomes: an LLM judge's verdict that the final answer is semantically equivalent to the ground truth, exact match against the canonical answer, a per-interaction bonus paid only for correct rollouts and capped once the annotated answer object has been rendered, and a penalty for malformed output. No term in the reward prescribes an intermediate viewpoint, action, or trajectory. The paper's headline evidence is that this suffices: average accuracy across six benchmarks reaches 53.2%, RL adds 4.6 points over the supervised initialization, gains over the base model grow from 0.6% on questions answerable from the initial view to 16.3% on questions requiring heavy exploration, and the learned behavior is adaptive -- searching for occluded targets, skipping needless moves, and gathering complementary viewpoints for spatial comparisons.

The premise it rests on. The load-bearing assumption is that the Qwen3-30B semantic-equivalence judge -- used both to supply the reinforcement-learning reward and to score every reported evaluation result -- agrees with human judgment about whether a short terse answer actually answers the question, and the paper never calibrates this judge against human raters, so if the judge is systematically lenient, both the RL training signal and the reported 53.2% accuracy are inflated.

If the paper is right (Pith's inferences, not the authors')

Outcome-based RL, with no viewpoint-level supervision anywhere in the reward, is sufficient to induce task-directed exploration: mixed-data RL raises average accuracy from 48.6% to 53.2% over the SFT initialization, improving five of six benchmarks.

Exploration demand governs how much the trained policy helps: gains over the base model climb from 0.6% on questions answerable from the initial view to 16.3% on the questions estimated to need the most interactions.

The policy spends its interaction budget adaptively rather than greedily: a flat per-interaction reward makes the model explore far more often (9.4-10.1 interactions per episode) but does not uniformly improve accuracy, slightly hurting in-domain benchmarks while helping held-out ones.

Single-view initialization is not a handicap: one rendered frame matches a bird's-eye-view-plus-four-frames input (41.0% vs 41.4% on ScanQA), and pose-consistent rendered observations beat nearest-view real-image retrieval by 10.6% on average.

The approach is input-efficient relative to prior open-source 3D methods, outperforming them while using 50-70% fewer frames.

Untested extension: the two-stage recipe is not tied to 3D scenes -- any interactive task with a verifiable outcome proxy (robot manipulation under partial view, surgical-image navigation, multi-page document question answering) could adopt it, but the paper demonstrates only indoor 3D rooms.

Testable prediction the paper does not run: because the relative action space is camera-centric and scale-invariant, the trained policy should transfer across scenes of very different physical sizes without retraining.

Transferable diagnostic: the paper's stratification of questions by geometry-estimated exploration demand could be applied to any embodied agent -- plot accuracy gain over a static-view baseline against predicted demand, and a healthy exploration policy shows gains that rise steeply with demand.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

We study how vision-language models (VLMs) can learn to think and actively explore in 3D space. Instead of providing multi-view images, videos, or 3D representations to the model, we preserve the native visual encoding of VLMs, i.e., a single image, one at a time, just like humans, and treat the 3D representation only as an environment with which the model can interact through actions (e.g., move left) and observations (rendered images). Each rollout begins from an initial first-person view and a visual question. The model can then iteratively reason, navigate, and reorient its viewpoint before committing to a final answer. Following an initial trajectory-supervised finetuning, we apply reinforcement learning (RL) with rewards driven primarily by final-answer correctness. Crucially, RL provides no direct supervision over which viewpoints the model should visit or which exploration actions it should take. Instead, the model must learn an effective exploration policy solely from the extent to which its interactions improve performance. Across six 3D benchmarks, our model ACT3D achieves an average accuracy of 53.2% that improves its base Qwen model by 22.0% and surpasses proprietary Gemini 3.6 Flash. ACT3D also outperforms existing open-source 3D methods while using 50-70% fewer input frames, and generalizes to unseen domains with an average improvement of 5.9% over its supervised initialization. Further analysis reveals adaptive, question-conditioned exploration. Performance gains over the base model increase from 0.6% on questions answerable from the initial view to 16.3% on those requiring extensive exploration. ACT3D searches for occluded targets, avoids unnecessary exploration, and acquires complementary viewpoints for spatial comparisons. These findings demonstrate that outcome-based RL enables VLMs not only to reason over visual evidence, but also to learn when and how to acquire it.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-16

The authors' main claim, as the classifier picked it out 1.00

“These findings demonstrate that outcome-based RL enables VLMs not only to reason over visual evidence, but also to learn when and how to acquire it.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.22 · human participants 0.03 · controlled comparison 0.92 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.99 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
We study how vision-language models (VLMs) can learn to think and actively explore in 3D space. contribution 0.79 0.56 0.05 asserted 0.49
Instead of providing multi-view images, videos, or 3D representations to the model, we preserve the native visual encoding of VLMs, i.e., a single image, one at a time, just like humans, and treat the 3D representation only as an environment with which the model can interact through actions (e.g., move left) and observations (rendered images). method 0.95 0.85 0.05 asserted 0.96
Each rollout begins from an initial first-person view and a visual question. method 1.00 0.70 0.02 asserted 0.99
The model can then iteratively reason, navigate, and reorient its viewpoint before committing to a final answer. method 0.99 0.60 0.05 asserted 0.52
Following an initial trajectory-supervised finetuning, we apply reinforcement learning (RL) with rewards driven primarily by final-answer correctness. method 1.00 0.95 0.04 asserted 1.00
Crucially, RL provides no direct supervision over which viewpoints the model should visit or which exploration actions it should take. method 0.93 0.89 0.06 asserted 1.00
Instead, the model must learn an effective exploration policy solely from the extent to which its interactions improve performance. method 0.97 0.70 0.12 asserted 0.99
Across six 3D benchmarks, our model ACT3D achieves an average accuracy of 53.2% that improves its base Qwen model by 22.0% and surpasses proprietary Gemini 3.6 Flash. result 1.00 0.97 0.15 asserted 1.00
ACT3D also outperforms existing open-source 3D methods while using 50-70% fewer input frames, and generalizes to unseen domains with an average improvement of 5.9% over its supervised initialization. result 1.00 0.96 0.09 asserted 1.00
Further analysis reveals adaptive, question-conditioned exploration. result 0.91 0.93 0.06 asserted 1.00
Performance gains over the base model increase from 0.6% on questions answerable from the initial view to 16.3% on those requiring extensive exploration. result 0.98 0.96 0.08 asserted 1.00
ACT3D searches for occluded targets, avoids unnecessary exploration, and acquires complementary viewpoints for spatial comparisons. result 0.85 0.90 0.06 asserted 1.00
These findings demonstrate that outcome-based RL enables VLMs not only to reason over visual evidence, but also to learn when and how to acquire it. interpretation 1.00 0.91 0.89 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.87

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-16: 1 request, 9,625 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (10)
Figure 1 of the paper
Figure 1. Learning to reason and explore in 3D. Unlike prior methods with passive perception which assumes receive complete 3D representations as input, Act3D treats 3D representations as interactive environments, learning to alternate between reasoning and actions to actively explore the 3D world before committing an answer. what information to acquire, how to interact with the environment, and how to reason over the accumulated evidence. To learn a policy that integrates visual information acquisition, spatial in￾teraction, and reasoning, we adopt a two-stage training procedure:…
Figure 2 of the paper
Figure 2. System and User Prompt template. See details in Appendix C.1
Figure 3 of the paper
Figure 3. Training data composition. Distribution of SFT trajectories and RL questions across
Figure 4 of the paper
Figure 4. Learning dynamics of Act3D. (a) ScanQA-only validation. (b) Mixed-data validation gains relative to SFT initialization. (c) Mean number of interactions for the final reward. Panel (c) is in-training rollout statistics; panels (a) and (b) are validation accuracies, and no value should be read across the two. RL substantially improves over SFT initialization. In the mixed-data setting, RL improves aver￾age accuracy from 48.6% to 53.2% (+4.6%) over the SFT init, with improvements on five of six benchmarks. The largest gains occur on ARKit (+14.1%) and SPBench-MV (+8.1%), highlight￾ing…
Figure 5 of the paper
Figure 5. Performance gains across different exploration demands. ScanQA validation questions are grouped by a geometry-based estimate of the interactions needed to reach the answer object from the initial viewpoint. Stacked bars show the accuracy improvements from SFT over the base model and subsequent RL over SFT. 4.4 QUALITATIVE ANALYSIS
Figure 6 of the paper
Figure 6. Qualitative analysis. We highlight emergent exploration behaviors. Arrows denote the executed actions, and text excerpts beneath illustrate the reasoning of the policy. render nearest photo +0.56 m, 26° off-axis render nearest photo +1.02 m, 25° off-axis render nearest photo +1.96 m, 24° off-axis
Figure 7 of the paper
Figure 7. Rendered versus retrieved observations. Comparison of pose-aligned renders and nearest
Figure 8 of the paper
Figure 8. ScanQA Acc. of different ac￾tion parameterizations. We investigate the effect of action parameterization by comparing relative camera movements with direct predic￾tion of absolute camera poses. As shown in
Figure 9 of the paper
Figure 9. Effect of supervised initialization on RL. We report ScanQA Acc. vs. average number of interactions per episode with and without SFT initialization. RL-only results are shown after 80 and 470 training steps. 4.6 ACTION SPACE 0 20 40 Accuracy (%) absolute relative 36.7 41.4
Figure 10 of the paper
Figure 10. Effect of interaction reward. ScanQA Acc. with and without the per-interaction reward across held-out and training benchmarks. Numbers below the bars indicate the average number of interactions per episode. serves primarily to encourage active evidence acquisition, while outcome-based feedback guides the policy toward effective reasoning. 5 CONCLUSION We presented Act3D, a framework for learning to think and explore in the 3D world by treating 3D representations as interactive environments rather than assumed complete inputs. Starting from a single observation, Act3D iteratively…

Figures are the authors', reproduced from the paper.

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment