Pith. sign in

REVIEW 4 major objections 6 minor 19 references

Structured physical memories let frozen video-language models check physical plausibility without any training.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 13:37 UTC pith:QDLHLQBS

load-bearing objection Solid training-free systems paper: hierarchical physical memory + graph rerank + rule-cue verification lifts several frozen VLMs on three hard plausibility benches; gains real but moderate and backbone-dependent. the 4 major comments →

arxiv 2607.10190 v1 pith:QDLHLQBS submitted 2026-07-11 cs.LG cs.AIcs.CV

PhysMRV: Physical Memory Retrieval and Verification for Physics Plausibility Reasoning

Wenyuan Wang , Lianyu Hu , Hao Wang , Yang Liu This is my paper
classification cs.LG cs.AIcs.CV
keywords physical plausibility reasoningvideo-language modelshierarchical memory bankphysics-aware retrievalevent graphstraining-free verificationfrozen VLMs
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Video-language models often recognize objects and actions yet still fail to judge whether an event obeys basic physical laws such as support, continuity, or contact. PhysMRV argues that the missing piece is not more fine-tuning but an external bank of structured physical precedents built from training videos. Each memory holds a scene caption, a physical-event graph of object interactions and state changes, and a rule-level summary with positive and negative diagnostic cues drawn from a fifteen-family physics catalog. At test time a coarse-to-fine retriever first recalls scenes by caption-question similarity, then reranks them by event-graph match and keeps a small set of rule-diverse anchors. A frozen model is then prompted to verify the current video against those retrieved rules and cues rather than answer from surface priors alone. Across three hard benchmarks and several strong backbones the method raises accuracy without updating any model parameters, and it measurably reduces the models’ strong bias toward calling every event plausible.

Core claim

A hierarchical Physical Memory Bank of captions, event graphs, and rule-cue records, retrieved by physics-aware coarse-to-fine matching and used as explicit verification evidence, consistently improves frozen VLMs’ physical-plausibility judgments over direct prompting on ImplausiBench, IntPhys2, and GRASP Level 2 without any parameter updates.

What carries the argument

Hierarchical Physical Memory Bank plus physics-aware coarse-to-fine retrieval: offline memories store scene captions, physical-event graphs, and rule-family cues; inference first recalls by caption-question embedding similarity, then reranks by weighted event-component match and selects rule-diverse top anchors that a frozen VLM uses as verification criteria.

Load-bearing premise

The event graphs and rule-cue records produced by the offline caption and language models must faithfully capture the true physical constraints, so that matching on them yields transferable physical precedents rather than merely similar-looking but physically mismatched cases.

What would settle it

Replace the event-graph reranker with pure visual or pure semantic retrieval while keeping the same verifier prompt and memory bank; if accuracy and negative-class detection no longer improve over direct prompting on GRASP Level 2 and IntPhys2, the claim that structured physical matching is necessary collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Frozen or proprietary VLMs can gain physical-plausibility skill at inference time without fine-tuning or access to model weights.
  • Retrieval for physical reasoning should be conditioned on interaction structure and rule families rather than surface semantic or visual similarity alone.
  • Explicit positive and negative diagnostic cues from matched precedents can correct the strong prior that frozen models treat almost every event as physically normal.
  • A reusable fifteen-family physics catalog plus event-graph memories can serve as a plug-and-play scaffold for other physical-reasoning benchmarks and backbones.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same hierarchical memory design could be applied to other latent-constraint domains where surface similarity is misleading, such as causal or counterfactual video reasoning.
  • Because gains depend on the backbone’s ability to use retrieved evidence, future work may need lighter adapters that help weaker instruction-following models actually read the verification cards.
  • Calibrated per-instance verification that grounds decisions in concrete object states and contacts observed in the current video remains the natural next bottleneck once memory retrieval is in place.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. PhysMRV is a training-free inference framework that improves frozen VLMs on physical-plausibility reasoning by converting TRAVL training videos into a Hierarchical Physical Memory Bank (scene captions, physical-event graphs, and rule-family principles with positive/negative cues), then retrieving rule-diverse precedents via caption–question semantic recall followed by event-graph component matching (Eqs. 5–6) and prompting a frozen VLM to verify the current video against those structured criteria. On ImplausiBench, IntPhys2, and GRASP Level 2, across Qwen3-VL, Cosmos-Reason2, InternVL3.5-8B, and LLaVA-OneVision-2, the full pipeline consistently beats direct prompting and component controls (Table 1), with ablations isolating matched retrieval, verifier context, top-1 vs top-3 diversity, and graph vs semantic/CLIP retrieval (Table 2), plus a class-conditional bias analysis (Table 3).

Significance. If the gains reflect transferable physical structure rather than correlated LLM physics language, the work is a useful systems contribution: a plug-and-play, non-parametric scaffold for physical plausibility that requires no parameter updates and is evaluated across three benchmarks and four backbones. Strengths include thorough component controls and retrieval-strategy ablations (Tables 1–2), explicit diagnosis of the severe positive-class prior in direct VLMs (Table 3), qualitative failure-mode corrections (Figure 4), and honest reporting that LLaVA-OneVision-2 barely benefits. The hierarchical memory design (caption / event graph / rule cues) and physics-aware coarse-to-fine retrieval are concrete and reusable. The result is incremental relative to RAG and physics-context builders, but the combination of structured physical memory with verification and bias analysis is a solid empirical step for frozen-VLM physical reasoning.

major comments (4)
  1. [§3.2–3.3, Eqs. (5)–(6), Table 2] §3.2–3.3 and Eqs. (5)–(6): Train and eval event graphs are both produced by Qwen3-VL-8B, and rule families/cues by GPT-5.5 from TRAVL VQA+explanations. Physical reranking therefore matches LLM-extracted structure to LLM-extracted structure. Table 2 shows graph reranking beats semantic/CLIP under the same verifier, which demonstrates internal pipeline consistency but not that graphs/cues are faithful carriers of ground-truth physical constraints independent of the verifier’s surface priors. Without human or physics-engine validation of graph/cue quality, or an ablation that swaps the extractor family (or uses human-authored graphs on a subset), the central claim that retrieval supplies transferable physical evidence remains under-supported relative to the alternative that the system retrieves self-consistent physics language and shifts answer polarity.
  2. [§4, Table 1, Memory Bank construction] §4 / Table 1: Memory is built exclusively from TRAVL training videos, while ImplausiBench is from the same TRAVL/ImplausiBench paper family (Motamed et al., 2025). The manuscript does not quantify scenario, template, or visual overlap between the 3,482 TRAVL train videos and ImplausiBench’s 150 paired scenarios. If memory anchors are near-neighbors of eval items, gains on ImplausiBench (and possibly shared physical templates elsewhere) may partly reflect in-distribution case retrieval rather than general physical verification. A contamination or nearest-neighbor analysis (e.g., max S_graph / caption similarity of eval items to the bank, and performance stratified by similarity) is needed for the cross-benchmark claim.
  3. [Table 3, §4.3, §5] Table 3 and §4.3–§5: The bias analysis is valuable and shows large negative-class gains, but on GRASP the correction is a large polarity rebalancing (Qwen3-VL 0.9453/0.0220 → 0.6636/0.4961; InternVL flips toward negative bias). Aggregate accuracy can rise from cue-driven prior correction without reliable per-instance physical verification. The conclusion itself flags calibrated instance-level verification as open. To make the central claim load-bearing, report error slices conditioned on retrieved rule-family match vs mismatch, and cases where retrieved cues are wrong yet the model still flips—or where graphs miss the true violation—so that gains cannot be explained solely as checklist-induced polarity shift.
  4. [§3.3, Eq. (5)] §3.3: Component weights α_p are fixed uniformly at 0.2, with top-50 recall and rule-diverse top-3 selection as free design choices. Sensitivity of S_graph and final accuracy to α_p, pool size, and diversity constraint is not reported. Because physical-event reranking is presented as the key differentiator from semantic RAG, at least a small sensitivity study (or learned/ablated weights) is needed to show that the reported gains are not brittle to these hyperparameters.
minor comments (6)
  1. [Throughout / §5] Naming is inconsistent: title/abstract use PhysMRV; body often uses PHYSMRV; conclusion says “training-only” while the rest says “training-free.” Unify terminology.
  2. [Table 1, §4.1] “LLaV A-OneVision-2” is repeatedly spaced incorrectly; fix to LLaVA-OneVision-2.
  3. [Figure 2, Implementation Details] Figure 2 lists “GPT 5.5”; implementation details say “GPT-5.5 API.” Clarify the exact model identifier and whether rule generation is reproducible (temperature, prompts).
  4. [§4 Metrics] ImplausiBench scoring is a custom option-letter parser without an official script (§4 Metrics). Release the scorer and parsing rules with the paper to make Table 1 reproducible.
  5. [§2] Related work on PhysMem, PCBs, and TRAVL is appropriate; a short explicit comparison table (training-free vs trained, memory structure, retrieval signal) would help position the contribution.
  6. [§3.1, §3.4] Eq. (9) writes ŷ_i = F_θ(P_i) while Eq. (4) includes (v_i, q_i, P_i). State clearly whether raw video frames are still passed to the verifier or only the structured prompt.

Circularity Check

0 steps flagged

No circular derivation: PhysMRV is an empirical retrieval-and-verification system evaluated on external benchmarks, not a first-principles claim that reduces to its inputs.

full rationale

This is a training-free systems paper. The Memory Bank is built offline from TRAVL training videos (captions/graphs via Qwen3-VL-8B, rules/cues via GPT-5.5), retrieval uses caption-question cosine recall then event-graph component matching (Eqs. 5–6), and a frozen VLM is prompted with retrieved rule-cue records plus the current caption/graph. Performance is reported as option/balanced accuracy against direct prompting and ablations on ImplausiBench, IntPhys2, and GRASP Level 2 (Tables 1–3). Nothing is fitted to the evaluation targets and then re-presented as a prediction; no uniqueness theorem or self-citation is load-bearing for a derived result; and the method does not redefine a quantity in terms of itself. Using TRAVL as non-parametric memory is standard RAG practice. Concerns about extractor–verifier correlation or train/eval overlap are correctness/generalization risks, not circularity of a claimed derivation. Score 0 with empty steps is the appropriate finding.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

The central claim rests on engineering choices and domain modeling assumptions, not on free physical constants. Load-bearing pieces are the hand-chosen retrieval hyperparameters, the 15-family rule taxonomy and LLM-generated graphs/cues, and the assumption that graph similarity tracks physical analogy closely enough for verification to help frozen VLMs.

free parameters (4)
  • component weights αp in event match M(e,e′) = 0.2 each
    Each of the five event components is weighted 0.2 by hand (§3.3); no sensitivity study is reported.
  • semantic recall pool size = 50
    Top-50 caption-question candidates kept before graph reranking; chosen, not derived.
  • final anchor count and diversity rule = top-3, one per rule family
    Greedy top-3 anchors forced into distinct rule families; design choice that affects reported Main results.
  • generation and decoding hyperparameters = as listed in Implementation Details
    fps=1.0, max tokens, temperature 0, video max pixels=151200 for graphs, etc., fix the memory content and answers.
axioms (5)
  • domain assumption Fifteen hand-defined physical rule families (support/gravity, force/motion, contact/collision, continuity/identity, fluids, thermal, optical, causal order, etc.) adequately cover the phenomena in the evaluation benchmarks.
    Catalog introduced in Figure 2 / §3.2 and used for diversity and cue generation; not validated as complete.
  • domain assumption Physical-event graphs extracted by prompting Qwen3-VL-8B (objects, state changes, evidence, phases) preserve the interaction structure needed for physical analogy.
    Graph construction Γ(v,q) is the backbone of reranking and verification (§3.2–3.3).
  • domain assumption A frozen VLM can act as a reliable visual verifier when given current caption/graph plus retrieved rule-cue cards, without parameter updates.
    Core of Rule-Cue Verification (§3.4); LLaVA results show this fails when the backbone ignores evidence.
  • domain assumption TRAVL training videos are a valid, non-leaking source of physical precedents for ImplausiBench, IntPhys2, and GRASP Level 2.
    Memory bank built exclusively from TRAVL (§3.1, Table 1 caption); separation from ImplausiBench is asserted but not proven in-text.
  • ad hoc to paper Cosine similarity of Qwen3-Embedding-0.6B on caption⊕question is a sufficient first-stage recall signal for physical tasks.
    Scene-level semantic retrieval stage (§3.3); alternative embeddings not compared.
invented entities (3)
  • Hierarchical Physical Memory Bank (caption / event-graph / rule-cue unit) no independent evidence
    purpose: Store reusable physical precedents for inference-time verification without updating the VLM.
    Defined in §3.1–3.2 as the core data structure; no independent existence outside this construction.
  • 15-family physical rule catalog with positive/negative cues no independent evidence
    purpose: Taxonomize principles and supply diagnostic checks for the verifier.
    Introduced in Figure 2; families and cues are LLM-generated for this paper.
  • Physics-aware coarse-to-fine retrieval (semantic recall + graph-component rerank + rule diversity) no independent evidence
    purpose: Select physically analogous rather than only semantically similar memories.
    Procedure and score Sgraph in §3.3 are paper-specific.

pith-pipeline@v1.1.0-grok45 · 18321 in / 3608 out tokens · 36052 ms · 2026-07-14T13:37:18.599172+00:00 · methodology

0 comments
read the original abstract

Video-language models (VLMs) have achieved remarkable performance on video understanding and visual question answering, yet they remain unreliable in reasoning about physical plausibility, where understanding object interactions, causal dynamics, and fundamental physical principles is essential. This limitation is particularly evident on challenging physical reasoning benchmarks, revealing a persistent gap in physical commonsense reasoning. To address this challenge, we propose PhysMRV, a training-free physical memory and verification framework for physical plausibility reasoning. Unlike retrieval-augmented VLMs that retrieve semantically similar videos as additional context, PhysMRV transforms training videos into a Hierarchical Memory Bank of structured physical knowledge comprising three complementary levels: scene descriptions capturing visual context, physical-event graphs modeling object interactions and causal structure, and physics-rule summaries distilling reusable physical principles and cues. During inference, PhysMRV retrieves physically relevant memories and leverages their structured physical evidence to guide a frozen VLM in verifying physical plausibility, requiring neither fine-tuning nor parameter updates. We evaluate PhysMRV on three challenging physical reasoning benchmarks, ImplausiBench, IntPhys2, and GRASP Level 2, across multiple state-of-the-art VLMs. Experimental results demonstrate consistent improvements over direct prompting across diverse VLMs and evaluation benchmarks, showing that structured physical memories provide an effective and scalable means of enhancing physical plausibility reasoning without additional training.

Figures

Figures reproduced from arXiv: 2607.10190 by Hao Wang, Lianyu Hu, Wenyuan Wang, Yang Liu.

Figure 1
Figure 1. Figure 1: Examples of physical plausibility reasoning across the evaluation benchmarks. Given short video evidence, a model must determine whether the observed event is physically plausible rather than relying only on object or action recognition. constraints. This limitation is also consistent with the observation in InPhyRe (Sreekumar & Boddeti, 2026), which suggests that physical reasoning relies on identifying v… view at source ↗
Figure 2
Figure 2. Figure 2: Structure of the Physics Memory Bank. Each training video contributes shared caption and physical-event rows, while question￾answer anchor rows store training questions, training labels, explanations, rule-family assignments, principles, and cue lists. Dense recall and physical reranking select training-only anchors for the verifier. retrieval-augmented approaches that primarily retrieve textual context to… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of PHYSMRV. Given a test video and question, PHYSMRV retrieves physically relevant training precedents from Physical Memory Bank via hierarchical retrieval: semantic similarity identifies candidate scenes, and physical-event matching selects the most event-relevant memories across diverse physical phenomena. A frozen VLM then uses the video’s extracted scene description and event graph to verify p… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative successful corrections. Left: a GRASP example where direct prompting follows the plausible/yes prior, while PHYSMRV uses retrieved continuity and force-change criteria to answer the correct “no.” Right: an IntPhys2 example where retrieved continuity and motion cues help change a direct possible answer to the correct impossible answer. three benchmarks—as representative probes of the failure mod… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 9 linked inside Pith

  1. [1]

    Llava-onevision-2: Towards next-generation perceptual intelligence

    An, X., Xie, Y., Tang, F., Yan, Y., Tan, H., Zhu, D., Chen, C., Zhao, X., Qin, B., Yang, K., Shen, Y., Zhang, Y., Zhang, K., Zhang, W., Cheng, Z., Zhang, N., Wu, C., Ge, C., Ran, Z., Song, D., Li, C., Feng, S., Hu, M., Chen, Z., Niu, J., Li, B., Feng, Z., Liu, Z., Ge, Z., and Deng, J. Llava-onevision-2: Towards next-generation perceptual intelligence. In ...

    2026

  2. [2]

    Bai, S. et al. Qwen3-VL technical report, 2025. URL https://arxiv.org/abs/2511.21631

    Pith/arXiv arXiv 2025

  3. [3]

    H., and Krishnan, R

    Balazadeh, V., Ataei, M., Cheong, H., Khasahmadi, A. H., and Krishnan, R. G. Physics context builders: A modular framework for physical reasoning in vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025. doi:10.1109/ICCV51701.2025.00687. URL https://arxiv.org/abs/2412.08619

    doi:10.1109/iccv51701.2025.00687 2025

  4. [4]

    T., Williams, A., Rabbat, M., and Dupoux, E

    Bordes, F., Garrido, Q., Kao, J. T., Williams, A., Rabbat, M., and Dupoux, E. IntPhys 2: Benchmarking intuitive physics understanding in complex synthetic environments, 2025. URL https://arxiv.org/abs/2506.09849

    Pith/arXiv arXiv 2025

  5. [5]

    Physgame: Uncovering physical commonsense violations in gameplay videos, 2024

    Cao, M., Tang, H., Zhao, H., Guo, H., Liu, J., Zhang, G., Liu, R., Sun, Q., Reid, I., and Liang, X. Physgame: Uncovering physical commonsense violations in gameplay videos, 2024. URL https://arxiv.org/abs/2412.01800

    Pith/arXiv arXiv 2024

  6. [6]

    PhysBench : Benchmarking and enhancing vision-language models for physical world understanding

    Chow, W., Mao, J., Li, B., Seita, D., Guizilini, V., and Wang, Y. PhysBench : Benchmarking and enhancing vision-language models for physical world understanding. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=Q6a9W6kzv5

    2025

  7. [7]

    GRASP : A novel benchmark for evaluating language grounding and situated physics understanding in multimodal language models, 2023

    Jassim, S., Holubar, M., Richter, A., Wolff, C., Ohmer, X., and Bruni, E. GRASP : A novel benchmark for evaluating language grounding and situated physics understanding in multimodal language models, 2023. URL https://arxiv.org/abs/2311.09048

    Pith/arXiv arXiv 2023

  8. [8]

    PhysMem : Scaling test-time memory for embodied physical reasoning, 2026

    Li, H., You, Y., Su, H., and Guibas, L. PhysMem : Scaling test-time memory for embodied physical reasoning, 2026. URL https://arxiv.org/abs/2602.20323

    Pith/arXiv arXiv 2026

  9. [9]

    Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. What makes good in-context examples for GPT-3 ? In Proceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, 2022. URL https://aclanthology.org/2022.deelio-1.10/

    2022

  10. [10]

    Video-RAG : Visually-aligned retrieval-augmented long video comprehension

    Luo, Y., Zheng, X., Li, G., Yin, S., Lin, H., Fu, C., Huang, J., Ji, J., Chao, F., Luo, J., and Ji, R. Video-RAG : Visually-aligned retrieval-augmented long video comprehension. In Advances in Neural Information Processing Systems, 2025. URL https://arxiv.org/abs/2411.13093

    arXiv 2025

  11. [11]

    BrainMem : Brain-inspired evolving memory for embodied agent task planning, 2026

    Ma, X., Hu, L., Tang, W., Hu, Z., Liao, Z., Wu, Z., and Liu, Y. BrainMem : Brain-inspired evolving memory for embodied agent task planning, 2026. URL https://arxiv.org/abs/2604.16331

    Pith/arXiv arXiv 2026

  12. [12]

    TRAVL : A recipe for making video-language models better judges of physics implausibility, 2025

    Motamed, S., Chen, M., Van Gool, L., and Laina, I. TRAVL : A recipe for making video-language models better judges of physics implausibility, 2025. URL https://arxiv.org/abs/2510.07550

    arXiv 2025

  13. [13]

    Cosmos-Reason2-8B

    NVIDIA . Cosmos-Reason2-8B . https://huggingface.co/nvidia/Cosmos-Reason2-8B, 2026. Hugging Face model card. Initial release: December 19, 2025; updated March 10, 2026. Accessed: 2026-06-30

    2026

  14. [14]

    Learning to retrieve prompts for in-context learning

    Rubin, O., Herzig, J., and Berant, J. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.\ 2655--2671, 2022. URL https://aclanthology.org/2022.naacl-main.191/

    2022

  15. [15]

    and Boddeti, V

    Sreekumar, G. and Boddeti, V. N. Inphyre discovers: Large multimodal models struggle in inductive physical reasoning, 2026. URL https://arxiv.org/abs/2509.12263

    Pith/arXiv arXiv 2026

  16. [16]

    Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency

    Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., Shao, J., et al. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    VideoTree : Adaptive tree-based video representation for LLM reasoning on long videos, 2024

    Wang, Z., Yu, S., Stengel-Eskin, E., Yoon, J., Cheng, F., Bertasius, G., and Bansal, M. VideoTree : Adaptive tree-based video representation for LLM reasoning on long videos, 2024. URL https://arxiv.org/abs/2405.19209

    Pith/arXiv arXiv 2024

  18. [18]

    M., Wang, Z., Yu, S., Bansal, M., and Bertasius, G

    Zhang, C., Lu, T., Islam, M. M., Wang, Z., Yu, S., Bansal, M., and Bertasius, G. A simple LLM framework for long-range video question-answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024. URL https://aclanthology.org/2024.emnlp-main.1209/

    2024

  19. [19]

    MemoryBank : Enhancing large language models with long-term memory

    Zhong, W., Guo, L., Gao, Q., Ye, H., and Wang, Y. MemoryBank : Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, pp.\ 19724--19731, 2024. doi:10.1609/aaai.v38i17.29946. URL https://doi.org/10.1609/aaai.v38i17.29946

    doi:10.1609/aaai.v38i17.29946 2024