Pith. sign in

REVIEW 4 major objections 7 minor 13 cited by

Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixels

T0 review · 4 major / 7 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A feed-forward network can recover a 3D scene's full material field—discrete type plus Young's modulus, Poisson's ratio, and density—from calibrated multi-view images, and then animate the scene with an MPM solver.

desk verdict A well-constructed pipeline that overclaims physical accuracy because its ground-truth labels are random draws from a VLM's ranges and its judge is the same model family. read the letter →

arxiv 2508.17437 v2 pith:CF4XOVQR submitted 2025-08-20 cs.CV

Long Le , Ryan Lucas , Chen Wang , Chuhao Chen , Dinesh Jayaraman , Eric Eaton , Lingjie Liu This is my paper
classification cs.CV
keywords 3DphysicspredictionmaterialfieldestimationCLIPfeaturedistillationsupervisedlearningGaussiansplattingPointMethodzero-shotsim-to-realPIXIEVERSEdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PIXIE argues that static visual appearance carries enough information to recover how an object will move and deform. To show this, it trains a feed-forward 3D U-Net to map CLIP-distilled volumetric features to a per-voxel material field, and it curates PIXIEVERSE, 1,624 synthetic objects with semi-automatically annotated material labels, to supply the supervision. If the central claim holds, then material acquisition stops being a slow per-scene optimization problem and becomes a single forward pass that takes about two seconds. The method reports realism scores 1.46–4.39x higher than test-time optimization baselines and transfers zero-shot to real scenes despite training only on synthetic data. The stakes are practical: fast, generalizable physics parameters would make reconstructed 3D scenes editable, interactive, and simulatable.

What carries the argument

The load-bearing object is the mapping fθ : (I, Π) → M̂ from calibrated multi-view RGB images to a voxelized material field M̂(p) = (ℓ̂, Ê, ν̂, d̂). To build it, PIXIE first distills CLIP embeddings into a NeRF feature field, voxelizes the field into a 64³×768 grid, projects it to 64 channels, and runs a 3D U-Net under a masked supervised loss (cross-entropy for material class, MSE for the three continuous parameters). The per-voxel mask, derived from NeRF density, prevents the network from collapsing onto the ~98% background voxels. The predicted field is transferred by nearest-neighbor interpolation onto Gaussian-splatting particles, which act as MPM particles.

What would settle it

Take held-out real objects, measure their Young's modulus and density by direct mechanical tests (e.g., stress-strain on a branch, weigh-and-volume density), then run PIXIE on photos of the same objects. If the predicted values are off by more than an order of magnitude while VLM realism scores stay high, the method is producing visually plausible but physically wrong fields.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that 3D appearance features distilled from a pretrained vision-language model—not RGB, not occupancy—are the right representation for supervised material prediction. PIXIE jointly predicts a discrete constitutive material model and continuous Young's modulus, Poisson's ratio, and density for every occupied voxel, then transfers these values to Gaussian-splatting particles for Material Point Method simulation. The authors claim this is the first generalizable feed-forward alternative to test-time physical parameter optimization: it produces full material fields in seconds, improves VLM-judged realism by 1.46–4.39x over DreamPhysics and OmniPhysGS, an

Load-bearing premise

The dataset labels are real physical quantities, not arbitrary random draws from ranges a vision-language model made up; if the labels are synthetic noise, the supervised loss teaches the network to imitate noise and the physical claims collapse.

Editorial extensions

If this is right

  • Physics parameters for an object can be inferred once from static multi-view images; no per-scene test-time optimization is needed, cutting preparation from hours to seconds.
  • The same trained network transfers to new synthetic scenes and, via CLIP's visual priors, to real-world scenes it never saw during training.
  • Predicted material fields plug directly into Gaussian-splatting-based MPM simulation, so static reconstructions become interactive worlds under gravity, wind, or user forces.
  • CLIP features are the load-bearing input: replacing them with RGB or occupancy features drops the realism score by 40–60% and roughly doubles parameter error.
  • A single inference jointly recovers material class, Young's modulus, Poisson's ratio, and density, unlike baselines that predict only a subset or require manual specification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's evaluation uses a VLM as judge; a strong VLM score shows visual plausibility but does not by itself establish that predicted Young's modulus or density matches measured physics. A lab-measurement comparison on real objects is the missing test.
  • Because dataset labels are drawn uniformly from VLM-specified ranges, retraining on a different random draw would reveal how much of the learned 'physics' is label noise; the paper's reported class accuracy and continuous MSE are errors against those same synthetic labels, not against measurements.
  • The point-estimate U-Net collapses each material range to a single value; a distributional or diffusion head (which the authors mention as future work) would better represent the inherent ambiguity the paper itself notes.
  • The same label-generation loop could be reused to auto-label larger 3D asset collections, using PIXIE's own predictions as priors; the paper does not claim this, but it follows from the semi-automatic VLM labeling pipeline.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. PIXIE proposes a feed-forward 3D U-Net that takes CLIP-distilled feature grids of a static scene (from posed RGB images) and predicts a per-voxel material field: a discrete material ID, Young's modulus, Poisson's ratio, and density. The network is trained with supervised losses (Eq. 2) on a new dataset, PIXIEVERSE, of 1,624 Objaverse objects whose material labels are produced by a semi-automatic VLM actor-critic pipeline with manually tuned in-context physics examples. The predicted field is transferred to Gaussian splatting and simulated with MPM. Experiments compare against DreamPhysics, OmniPhysGS, and NeRF2Physics using Gemini-2.5-Pro as a realism judge, and report VLM scores, PSNR/SSIM, and material accuracy; zero-shot generalization to real scenes is shown qualitatively. The paper claims 1.46-4.39x better realism and orders-of-magnitude faster inference than test-time optimization baselines.

Significance. If the core claim were supported, this would be a meaningful contribution: a fast, generalizable feed-forward estimator of material fields from static images, coupled with a large open dataset, would be of value to simulation and 3D content creation. The paper also deserves credit for clearly describing the data pipeline and ablating CLIP vs RGB vs occupancy features. However, the central claim that the network learns physical parameters is not established. The continuous training targets are uniform draws from VLM-proposed ranges (Appendix B.4), not measurements; the evaluation judge is from the same model family that generated the labels (Section 4, Appendix D); and no class-mean baseline is compared. The reported continuous errors and VLM gains may therefore reflect reproduction of pseudo-label priors rather than physically grounded prediction. The real-scene generalization evidence is only qualitative. The speed advantage is real, but the '1.46-4.39x better' quality claim is uninterpretable as physical accuracy.

major comments (4)
  1. [Appendix B.4; Eq. (2)] The continuous labels are not physical measurements. The paper states: 'we prompt the VLM actor to output a plausible range ... We then sample a value uniformly from each range to build our training dataset.' Thus Eq. (2) regresses on random draws from VLM-specified intervals. For a fixed semantic part, the Bayes-optimal MSE predictor is the mean of that interval; given the reported 0.985 material-class accuracy, the network may simply be outputting class-conditional means. No per-class mean/median baseline is reported. Consequently, the 'Avg. Cont. MSE,' and per-parameter errors in Table 1 do not measure physical accuracy; they measure deviation from synthetic targets.
  2. [Section 4; Appendix D] The main evaluation is circular. Gemini-2.5-Pro generates the dataset labels (via the VLM actor in Appendix B.4) and is also the judge that scores the candidate simulation videos (System Prompt in Appendix D). VLM scores therefore reward consistency with the labeler's priors, not physical plausibility. The '1.46-4.39x better' claim in Table 1 and Figure 4 is thus not anchored to any external ground truth. The PSNR/SSIM numbers are computed against 'reference videos in PIXIEVERSE' that are themselves rendered from the same pseudo-labeled data, so they do not break the circularity.
  3. [Section 4.2; Figure 6] Zero-shot generalization to real scenes is only qualitative. Figure 6 shows predicted material types and Young's modulus for a few real scenes, with no quantitative metric, no human study, and no comparison to any baseline under this setting. The statements 'correctly assigns rigid vase bases and flexible leaves' and 'surprisingly performant' are anecdotal. Without measurable evidence (e.g., trajectory agreement, human preference scores, or comparison against simple appearance heuristics), the sim-to-real claim is not supported.
  4. [Section 3.2; Table 2] The discrete material labels are also VLM-generated and propagated through CLIP feature similarity, so they inherit the same prior. The dataset ablation in Table 2 shows that in-context examples change VLM scores from 1.34 to 4.83, which is consistent with the judge and labeler sharing priors. This reinforces, rather than resolves, the concern that the evaluation measures the labeler's consistency. Independent validation (e.g., real object drop tests, force-displacement measurements, or a human physical-plausibility study) is needed before 'physics' can be claimed.
minor comments (7)
  1. [Abstract; Section 1] The phrase 'physic material annotations' should be 'physical material annotations.'
  2. [Section 3.1] Typo 'visual featutes' in the paragraph after Eq. (1).
  3. [Figure 10 caption] 'high-performant' should be 'high-performing.'
  4. [Appendix E.2] The input feature notation 'F ∈ R N ×3×D×H×W' is confusing; should presumably be N × C × 64 × 64 × 64 with C=3, 768, or 1. Please clarify the channel/spatial ordering.
  5. [Appendix E.3] The U-Net is cited as '[9, 34]'; [9] is Dhariwal and Nichol (diffusion models), which is not the standard U-Net reference. The citation should be [34] (Ronneberger) or an appropriate follow-up.
  6. [Section 4; Table 1] The paper does not describe how the 'reference videos in PIXIEVERSE' are generated or how the synthetic test scenes are split. Please provide dataset split details and a precise definition of the PSNR/SSIM reference.
  7. [Appendix D; Table 2] 'executionability rate' is a nonstandard term; consider 'successful execution rate' or 'numerical stability rate.'

Circularity Check

2 steps flagged · score 7.0 of 10

PIXIE's continuous 'ground-truth' parameters are uniform samples from Gemini-proposed ranges, and the realism judge is the same Gemini-2.5-Pro model, so the main quality claims reduce to agreement with the VLM labeler rather than external physics.

  1. self definitional [Appendix B.4 (VLM Actor-Critic Labeling), used by Eq. 2 in Sec. 3.1]
    "Additionally, materials in the real-world contain uncertainty that visual information alone cannot resolve (e.g., a tree can range from stiff to flexible). Thus, instead of specifying one physics parameter per part, we prompt the VLM actor to output a plausible range (e.g., E ∈ {1e4, 1e5} see Fig. 11, 12). We then sample a value uniformly from each range to build our training dataset."

    Eq. 2 trains the network against E_GT, nu_GT, and d_GT, but these targets are uniform random draws from a Gemini-specified interval, not measured material constants. For any recognized semantic part, the MSE-optimal continuous prediction is the mean of that sampled range, so Table 1's 'Avg. Cont. MSE' and per-parameter errors mostly measure how well the network identifies parts and outputs class-conditional range means. The paper never compares against such a class-mean baseline; the 'prediction of physics' is therefore statistically forced toward reproducing the VLM's plausible ranges rather than recovering physical parameters.

  2. other [Sec. 4 'Evaluation Metrics' and Appendix D]
    "We utilize a state-of-the-art vision-language model, Gemini-2.5-Pro [36] as the judge. The models are prompted to compare the rendered candidate animations generated using physics parameters predicted by different baselines, and score those videos on a scale from 0 to 5, where a higher score is better."

    The same Gemini-2.5-Pro model that, in Appendix B.4, proposed the ranges from which PIXIE's training labels were sampled is used as the realism judge. PIXIE is explicitly trained to regress onto those Gemini-generated labels, so a high Gemini score is by construction expected when the simulated videos resemble the labeler's prior. The reported 1.46-4.39x 'realism' improvement therefore measures agreement between the predictor and the label-generating model, not independent physical fidelity. The speed advantage remains genuine, but the headline quality comparison is an evaluation loop.

full rationale

The forward pipeline (CLIP feature field -> 3D U-Net -> material grid) is internally consistent, and Eq. 2 is a standard supervised loss with no algebraic self-reference; there is no load-bearing self-citation chain. The circularity is in the dataset/evaluation loop: the continuous labels are uniform samples from Gemini-proposed ranges and are explicitly called 'ground-truth,' while the quantitative realism metric is scored by the same Gemini-2.5-Pro family. Consequently, the central claim that PIXIE learns physical parameters from images and is 1.46-4.39x better in realism reduces, for the continuous parameters, to range-mean regression and, for the realism score, to agreement with the model that generated the labels. The paper's own limitation statement concedes that visual information alone cannot resolve the sampled stiffness uncertainty, underscoring that the pseudo-labels are not determined by the input. A class-mean or part-classification baseline is never provided, leaving the reported continuous accuracy and VLM realism scores unanchored to external physics. The speed and zero-shot qualitative demonstrations are real and non-circular, so the score is substantial but not maximal.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central contribution rests on a pseudo-labeled dataset constructed from manually tuned VLM ranges, and no external physical measurements are used. The free parameters are mostly human-chosen ranges and architecture choices; no new physical entities are postulated.

free parameters (6)
  • Per-class physics ranges (E, Poisson, density) = e.g., tree E in [1e4, 1e5] Pa, density [200, 400] kg/m3; soda can E in [5e10, 8e10] Pa; rubber duck E in [3e4, 5e4] Pa
    Manually tuned by the authors (Fig. 12, Listing 1) and used to define the training labels; these ranges, not measurements, determine the ground truth.
  • Occupancy threshold alpha = 0.01
    Filters background voxels before computing the supervised loss in Eq. 2.
  • Loss balancing factor lambda = not reported
    Weights the cross-entropy term against the MSE terms in Eq. 2.
  • Voxel grid size N = 64
    Resolution of the feature and material grids; an architecture choice that limits spatial detail.
  • CLIP feature dimension D = 768
    Input feature dimension; main text projects to 64 dims, Appendix E.2 says 32 channels.
  • Normalization min/max statistics for log E, nu, log rho = computed from PIXIEVERSE
    Values are normalized to [-1, 1] before training and evaluation, which changes the meaning of reported MSE.
assumptions (5)
  • domain assumption CLIP-distilled volumetric features encode sufficient information to estimate physical parameters
    Central thesis in Sec 3; not independently tested, and the paper's own ablations only compare CLIP against RGB and occupancy features.
  • ad hoc to paper Uniform samples from VLM-proposed ranges are valid physical ground truth
    Appendix B.4 samples values from ranges; if false, the supervised regression does not teach physics.
  • ad hoc to paper Gemini-2.5-Pro realism scores are a valid measure of physical plausibility
    Sec 4 and App D use Gemini as judge, and the same model family generated the labels, creating a self-referential evaluation.
  • domain assumption NeRF density thresholding provides correct occupancy for supervision
    Sec 3.1 constructs the occupancy mask M from NeRF densities above alpha; incorrect occupancy would misdirect the masked loss.
  • domain assumption The PhysGaussian MPM solver faithfully realizes the predicted material fields
    Sec A.3 and Sec 4 rely on an existing MPM implementation; solver artifacts are not separated from prediction error.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixels." pith.science (2026). https://pith.science/paper/CF4XOVQR

@misc{pith2026250817437,
  author       = {Pith},
  title        = {Pith review of: Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixels},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CF4XOVQR}},
  note         = {Machine review of arXiv:2508.17437}
}
read the original abstract

Inferring the physical properties of 3D scenes from visual information is a critical yet challenging task for creating interactive and realistic virtual worlds. While humans intuitively grasp material characteristics such as elasticity or stiffness, existing methods often rely on slow, per-scene optimization, limiting their generalizability and application. To address this problem, we introduce PIXIE, a novel method that trains a generalizable neural network to predict physical properties across multiple scenes from 3D visual features purely using supervised losses. Once trained, our feed-forward network can perform fast inference of plausible material fields, which coupled with a learned static scene representation like Gaussian Splatting enables realistic physics simulation under external forces. To facilitate this research, we also collected PIXIEVERSE, one of the largest known datasets of paired 3D assets and physic material annotations. Extensive evaluations demonstrate that PIXIE is about 1.46-4.39x better and orders of magnitude faster than test-time optimization methods. By leveraging pretrained visual features like CLIP, our method can also zero-shot generalize to real-world scenes despite only ever been trained on synthetic data. https://pixie-3d.github.io/

Discussion (0). Sign in to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PhysInOne: Visual Physics Learning and Reasoning in One Suite

    cs.CV 2026-04 unverdicted novelty 8.0 of 10

    PhysInOne is a new dataset of 2 million videos across 153,810 dynamic 3D scenes covering 71 physical phenomena, shown to improve AI performance on physics-aware video generation, prediction, property estimation, and m...

  2. 3D Generation for Embodied AI and Robotic Simulation: A Survey

    cs.RO 2026-04 accept novelty 7.0 of 10

    3D generation for embodied AI is shifting from visual realism toward interaction readiness, organized into data generation, simulation environments, and sim-to-real bridging roles.

  3. PerpetualWonder: Long-Horizon Action-Conditioned 4D Scene Generation

    cs.CV 2026-02 unverdicted novelty 7.0 of 10

    PerpetualWonder introduces a closed-loop generative simulator with a unified physical-visual representation for long-horizon action-conditioned 4D scene generation from one image.

  4. UniPixie: Unified and Probabilistic 3D Physics Learning via Flow Matching

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    UniPixie learns a parameterized continuous path of material properties from images via flow matching, producing simulation-ready outputs for multiple physics solvers and cutting Young's modulus error by over 50%.

  5. NeuROK: Generative 4D Neural Object Kinematics

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    NeuROK learns a data-driven latent kinematic parameterization on a large 4D dataset to generate realistic object deformations by simulating dynamics only in low-dimensional latent space via Lagrangian mechanics.

  6. PhysX-Omni: Unified Simulation-Ready Physical 3D Generation for Rigid, Deformable, and Articulated Objects

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    PhysX-Omni unifies simulation-ready 3D asset generation across rigid, deformable, and articulated objects via a new geometry representation, the PhysXVerse dataset, and the PhysX-Bench evaluation suite.

  7. Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    The paper proposes a problem-driven taxonomy for feed-forward 3D scene modeling that groups methods by five core challenges: feature enhancement, geometry awareness, model efficiency, augmentation strategies, and temp...

  8. VDAWorld: World Modelling via VLM-Directed Abstraction and Simulation

    cs.CV 2025-12 conditional novelty 6.0 of 10

    A vision-language model writes a simulation program—grounded by segmentation and 3D tools—that predicts physically plausible futures from an image and caption, outperforming video generators on modified benchmarks.

  9. PhysChoreo: Physics-Controllable Video Generation with Part-Aware Semantic Grounding

    cs.CV 2025-11 conditional novelty 6.0 of 10

    A two-stage framework that predicts per-part material properties from a single image and uses editable physics simulation to guide video generation.

  10. Physically Interpretable World Models via Weakly Supervised Representation Learning

    cs.LG 2024-12 unverdicted novelty 6.0 of 10

    PIWM aligns latent states in image-based world models with physical variables and constrains their dynamics to known equations via weak distribution supervision, yielding accurate long-horizon predictions and paramete...

  11. Automatically Improving Simulation Physics for Articulated Objects

    cs.RO 2026-05 unverdicted novelty 5.0 of 10

    A simulator-in-the-loop multi-modal method refines physical properties of incomplete 3D articulated objects to improve simulation stability and downstream robot policy performance.

  12. 3D Generation for Embodied AI and Robotic Simulation: A Survey

    cs.RO 2026-04 unverdicted novelty 3.0 of 10

    The survey organizes 3D generation for embodied AI into data generators for assets, simulation environments for interaction, and sim-to-real bridges, noting a shift toward interaction readiness and listing bottlenecks...

  13. 3D Generation for Embodied AI and Robotic Simulation: A Survey

    cs.RO 2026-04 unverdicted novelty 2.0 of 10

    The paper surveys 3D generation techniques for embodied AI and robotics, categorizing them into data generation, simulation environments, and sim-to-real bridging while identifying bottlenecks in physical validity and...

Reference graph

Works this paper leans on

50 extracted references · 25 canonical work pages · cited by 11 Pith papers

  1. [1]

    Physically embodied gaussian splatting: A realtime correctable world model for robotics

    Jad Abou-Chakra, Krishan Rana, Feras Dayoub, and Niko Suenderhauf. Physically embodied gaussian splatting: A realtime correctable world model for robotics. In 8th Annual Conference on Robot Learning, 2024. URL https://openreview.net/forum?id=AEq0onGrN2

    work page 2024

  2. [2]

    Physion: Evaluating physical prediction from vision in humans and machines

    Daniel M Bear, Elias Wang, Damian Mrowca, Felix J Binder, Hsiao-Yu Fish Tung, RT Pramod, Cameron Holdaway, Sirui Tao, Kevin Smith, Fan-Yun Sun, et al. Physion: Evaluating physical prediction from vision in humans and machines. arXiv preprint arXiv:2106.08261, 2021

    arXiv 2021

  3. [3]

    Material recognition in the wild with the materials in context database

    Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. Material recognition in the wild with the materials in context database. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3479–3487, 2015

    work page 2015

  4. [4]

    Physx: Physical-grounded 3d asset generation

    Ziang Cao, Zhaoxi Chen, Liang Pan, and Ziwei Liu. Physx: Physical-grounded 3d asset generation. arXiv preprint arXiv:2507.12465, 2025

    arXiv 2025

  5. [5]

    Physgen3d: Crafting a miniature interactive world from a single image

    Boyuan Chen, Hanxiao Jiang, Shaowei Liu, Saurabh Gupta, Yunzhu Li, Hao Zhao, and Shenlong Wang. Physgen3d: Crafting a miniature interactive world from a single image. arXiv preprint arXiv:2503.20746, 2025

    arXiv 2025

  6. [6]

    Vid2sim: Generalizable, video-based reconstruction of appearance, geometry and physics for mesh-free simulation

    Chuhao Chen, Zhiyang Dou, Chen Wang, Yiming Huang, Anjun Chen, Qiao Feng, Jiatao Gu, and Lingjie Liu. Vid2sim: Generalizable, video-based reconstruction of appearance, geometry and physics for mesh-free simulation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    work page 2025

  7. [7]

    Unireal: Universal image generation and editing via learning real-world dynamics

    Xi Chen, Zhifei Zhang, He Zhang, Yuqian Zhou, Soo Ye Kim, Qing Liu, Yijun Li, Jianming Zhang, Nanxuan Zhao, Yilin Wang, et al. Unireal: Universal image generation and editing via learning real-world dynamics. arXiv preprint arXiv:2412.07774, 2024

    arXiv 2024

  8. [8]

    Objaverse: A universe of annotated 3d objects, 2022

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects, 2022. URL https://arxiv.org/abs/2212.08051

    arXiv 2022

Show all 50 references
  1. [9]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021

    2021

  2. [10]

    Pie-nerf: Physics-based interactive elastodynamics with nerf, 2023

    Yutao Feng, Yintong Shang, Xuan Li, Tianjia Shao, Chenfanfu Jiang, and Yin Yang. Pie-nerf: Physics-based interactive elastodynamics with nerf, 2023

    2023

  3. [11]

    Sama: Material-aware 3d selection and segmentation

    Michael Fischer, Iliyan Georgiev, Thibault Groueix, Vladimir G Kim, Tobias Ritschel, and Valentin Deschaintre. Sama: Material-aware 3d selection and segmentation. arXiv preprint arXiv:2411.19322, 2024

    2024

  4. [12]

    Tenenbaum, Kaiming He, and Wojciech Matusik

    Minghao Guo, Bohan Wang, Pingchuan Ma, Tianyuan Zhang, Crystal Elaine Owens, Chuang Gan, Joshua B. Tenenbaum, Kaiming He, and Wojciech Matusik. Physically compatible 3d object modeling from a single image. arXiv preprint arXiv:2405.20510, 2024

    2024 arXiv

  5. [13]

    Autovfx: Physically realistic video editing from natural language instructions

    Hao-Yu Hsu, Zhi-Hao Lin, Albert Zhai, Hongchi Xia, and Shenlong Wang. Autovfx: Physically realistic video editing from natural language instructions. arXiv preprint arXiv:2411.02394, 2024

    2024 arXiv

  6. [14]

    Dreamphysics: Learning physical properties of dynamic 3d gaussians with video diffusion priors.arXiv preprint arXiv:2406.01476, 2024

    Tianyu Huang, Yihan Zeng, Hui Li, Wangmeng Zuo, and Rynson WH Lau. Dreamphysics: Learning physical properties of dynamic 3d gaussians with video diffusion priors.arXiv preprint arXiv:2406.01476, 2024

    2024 arXiv

  7. [15]

    gradsim: Differentiable simulation for system identification and visuomotor control

    Krishna Murthy Jatavallabhula, Miles Macklin, Florian Golemo, Vikram V oleti, Linda Petrini, Martin Weiss, Breandan Considine, Jerome Parent-Levesque, Kevin Xie, Kenny Erleben, Liam Paull, Florian Shkurti, Derek Nowrouzezahrai, and Sanja Fidler. gradsim: Differentiable simulat...

    2021

  8. [16]

    Phystwin: Physics-informed reconstruction and simulation of deformable objects from videos

    Hanxiao Jiang, Hao-Yu Hsu, Kaifeng Zhang, Hsin-Ni Yu, Shenlong Wang, and Yunzhu Li. Phystwin: Physics-informed reconstruction and simulation of deformable objects from videos. arXiv preprint arXiv:2503.17973, 2025

    2025 arXiv

  9. [17]

    Cotracker: It is better to track together

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. In European Conference on Computer Vision, pages 18–35. Springer, 2024

    2024

  10. [18]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139–1, 2023

    2023

  11. [19]

    Lerf: Language embedded radiance fields

    Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19729–19739, 2023

    2023

  12. [20]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  13. [21]

    Decomposing nerf for editing via feature field distillation

    Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. In Advances in Neural Information Processing Systems, volume 35,

  14. [22]

    Articulate-anything: Automatic modeling of articulated objects via a vision-language foundation model

    Long Le, Jason Xie, William Liang, Hung-Ju Wang, Yue Yang, Yecheng Jason Ma, Kyle Vedder, Arjun Krishna, Dinesh Jayaraman, and Eric Eaton. Articulate-anything: Automatic modeling of articulated objects via a vision-language foundation model. arXiv preprint arXiv:2410.13882, 2024

    2024 arXiv

  15. [23]

    PAC-neRF: Physics augmented continuum neural radiance fields for geometry-agnostic system identification

    Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chen- fanfu Jiang, and Chuang Gan. PAC-neRF: Physics augmented continuum neural radiance fields for geometry-agnostic system identification. In The Eleventh International Conference on Learn- in...

    2023

  16. [24]

    Generative image dynamics

    Zhengqi Li, Richard Tucker, Noah Snavely, and Aleksander Holynski. Generative image dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24142–24153, 2024

    2024

  17. [25]

    Wonderplay: Dynamic 3d scene generation from a single image and actions

    Zizhang Li, Hong-Xing Yu, Wei Liu, Yin Yang, Charles Herrmann, Gordon Wetzstein, and Jiajun Wu. Wonderplay: Dynamic 3d scene generation from a single image and actions. arXiv preprint arXiv:2505.18151, 2025

    2025

  18. [26]

    OmniphysGS: 3d constitutive gaussians for general physics-based dynamics generation

    Yuchen Lin, Chenguo Lin, Jianjin Xu, and Yadong MU. OmniphysGS: 3d constitutive gaussians for general physics-based dynamics generation. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=9HZtP6I5lv

    2025

  19. [27]

    Learning neural constitutive laws from motion observations for generalizable pde dynamics

    Pingchuan Ma, Peter Yichen Chen, Bolei Deng, Joshua B Tenenbaum, Tao Du, Chuang Gan, and Wojciech Matusik. Learning neural constitutive laws from motion observations for generalizable pde dynamics. In International Conference on Machine Learning, pages 23279–23300. PMLR, 2023

    2023

  20. [28]

    Nerf: Representing scenes as neural radiance fields for view synthesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021

    2021

  21. [29]

    Uniphy: Learning a unified constitutive model for inverse physics simulation

    Himangi Mittal, Peiye Zhuang, Hsin-Ying Lee, and Shubham Tulsiani. Uniphy: Learning a unified constitutive model for inverse physics simulation. arXiv preprint arXiv:2505.16971, 2025

    2025 arXiv

  22. [30]

    Genie 2: A large-scale foundation world model

    Jack Parker-Holder, Philip Ball, Jake Bruce, Vibhavari Dasagi, Kristian Holsheimer, Christos Kaplanis, Alexandre Moufarek, Guy Scully, Jeremy Shar, Jimmy Shi, Stephen Spencer, Jessica Yung, Michael Dennis, Sultan Kenjeyev, Shangbang Long, Vlad Mnih, Harris Chan, Maxime Gazeau,...

    2024

  23. [31]

    D-NeRF: Neural Radiance Fields for Dynamic Scenes

    Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020

    2020

  24. [32]

    Feature splatting: Language-driven physics-based scene synthesis and editing

    Ri-Zhao Qiu, Ge Yang, Weijia Zeng, and Xiaolong Wang. Feature splatting: Language-driven physics-based scene synthesis and editing. arXiv preprint arXiv:2404.01223, 2024

    2024 arXiv

  25. [33]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  26. [34]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015

    2015

  27. [35]

    Distilled feature fields enable few-shot language-guided manipulation, 2023

    William Shen, Ge Yang, Alan Yu, Jansen Wong, Leslie Pack Kaelbling, and Phillip Isola. Distilled feature fields enable few-shot language-guided manipulation, 2023. URL https: //arxiv.org/abs/2308.07931

    2023 arXiv

  28. [36]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  29. [37]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35:10078–10093, 2022

    2022

  30. [38]

    Physctrl: Generative physics for controllable and physics-grounded video generation

    Chen Wang, Chuhao Chen, Yiming Huang, Zhiyang Dou, Yuan Liu, Jiatao Gu, and Lingjie Liu. Physctrl: Generative physics for controllable and physics-grounded video generation. In arXiv preprint, 2025

    2025

  31. [39]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems, 33:5776–5788, 2020

    2020

  32. [40]

    Video2game: Real-time, interactive, realistic and browser-compatible environment from a single video, 2024

    Hongchi Xia, Zhi-Hao Lin, Wei-Chiu Ma, and Shenlong Wang. Video2game: Real-time, interactive, realistic and browser-compatible environment from a single video, 2024

    2024

  33. [41]

    Physgaussian: Physics-integrated 3d gaussians for generative dynamics

    Tianyi Xie, Zeshun Zong, Yuxing Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. arXiv preprint arXiv:2311.12198, 2023

    2023 arXiv

  34. [42]

    Physical property understanding from language-embedded feature fields

    Albert J Zhai, Yuan Shen, Emily Y Chen, Gloria X Wang, Xinlei Wang, Sheng Wang, Kaiyu Guan, and Shenlong Wang. Physical property understanding from language-embedded feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 282...

    2024

  35. [43]

    Particle-grid neural dynamics for learning deformable object models from rgb-d videos

    Kaifeng Zhang, Baoyu Li, Kris Hauser, and Yunzhu Li. Particle-grid neural dynamics for learning deformable object models from rgb-d videos. arXiv preprint arXiv:2506.15680, 2025

    2025

  36. [44]

    Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, and William T

    Tianyuan Zhang, Hong-Xing Yu, Rundi Wu, Brandon Y . Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, and William T. Freeman. PhysDreamer: Physics-based interaction with 3d objects via video generation. In European Conference on Computer Vision. Springer, 2024

    2024

  37. [45]

    tree", “rubber toy

    Licheng Zhong, Hong-Xing Yu, Jiajun Wu, and Yunzhu Li. Reconstruction and simulation of elastic objects with spring-mass 3d gaussians. European Conference on Computer Vision (ECCV), 2024. 12 A Preliminaries This section briefly reviews foundational concepts in 3D scene represe...

    2024

  38. [47]

    Specifically, ur VLM actor proposes segmentation queries based on a set of mutli-view images of the object as shown in Fig

    We use VLM to propose object-dependent segmentation while NeRF2Physics using LLM is essentially blind. Specifically, ur VLM actor proposes segmentation queries based on a set of mutli-view images of the object as shown in Fig. 11

  39. [48]

    pot", "trunk

    We use semantic proposals (e.g., "pot", "trunk") instead of material proposals (e.g., "leather", "stone") like NeRF2Physics did. Computing similarity directly between material name and CLIP features yields inaccurate and noisy segmentation as shown in Fig. 10. This also limits...

  40. [49]

    pot,leaves

    We proposes multiple candidates (e.g., "pot,leaves" vs "base,folliage") and use a VLM critic to select the best based on CLIP-based segmentation while NeRF2Physics does not have any selection mechanism. Figure 10 show the dramatic segmentation quality across different queries,...

  41. [50]

    tree swaying in the wind

    We also provide manually tuned in-context physics parameter examples. These crucial differences contribute to much higher quality dataset labeling as shown in Tab. 2. D VLM As a Physics Judge We utilize a VLM to evaluate the realism of different candidate videos. The videos ar...

  42. [2022]

    URL https://arxiv.org/pdf/2205.15585.pdf

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.