Pith. sign in

REVIEW 4 major objections 6 minor 24 references

Any image, video, or text can be represented as a bag of canonical atomic propositions, enabling compositional retrieval and coverage analysis that dense embeddings cannot match.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Multimodal observations are represented as bags of canonical atomic propositions, enabling compositional retrieval and corpus coverage analysis on driving and open-world video data.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection A well-motivated pipeline for interpretable multimodal representation, but the retrieval claim rests on an evaluation that doesn't actually test it. the 4 major comments →

arxiv 2607.16560 v1 pith:2UCQLK35 submitted 2026-07-18 cs.AI cs.CVcs.LGcs.MM

From Modalities to Propositions: A Language-Centric Framework for Multimodal Intelligence

Nadine Chang , Maying Shen , Shizhe Diao , Jialiang Wang , Jingde Chen , Thomas Breuel , Pavlo Molchanov , Rafid Mahmood
show 1 more author
Jose M. Alvarez
This is my paper
classification cs.AI cs.CVcs.LGcs.MM
keywords multimodal representationatomic propositionssemantic codebookcompositional retrievalcorpus coverage analysisautonomous drivinginterpretability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes treating every multimodal observation—an image, a video clip, or a text log—as a bag of atomic propositions: short factual statements such as 'pedestrian crossing street' or 'traffic light is red'. A global semantic codebook collapses paraphrases into a single canonical form, so images, video, and text end up in one shared, human-readable vocabulary. The authors argue that this representation supports compositional reasoning (complex queries built by combining facts), interpretable retrieval, and corpus coverage analysis that exposes concrete train-versus-test knowledge gaps in autonomous driving and open-world video data. If the claim holds, it gives practitioners an inspectable layer over large multimodal corpora, complementing rather than replacing opaque dense embeddings.

Core claim

The paper's central claim is that the semantic content of any observation can be captured by a set of atomic propositions, each an elementary statement of the form (relation, arguments) that cannot be split without losing meaning. Working from a natural-language description produced by a multimodal large language model, the framework distills sentences, converts them into propositions, filters task-irrelevant modifiers, and then maps each proposition to a canonical form in a corpus-level semantic codebook. The codebook is induced by ranking propositions by frequency, gathering paraphrases around frequent leaders using embedding similarity, and verifying semantic equivalence with an LLM. The

What carries the argument

The central object is the Bag of Atomic Propositions (BoAP): a set of elementary factual statements, each written as a relation with noun-anchored arguments (for example, 'pedestrian crossing street'), replacing the observation itself for semantic purposes. The companion object is the global semantic codebook, a corpus-derived vocabulary of canonical propositions built by ranking all extracted propositions by frequency, grouping paraphrases around each leader via embedding cosine similarity, and then asking an LLM to verify semantic equivalence before keeping the leader as the canonical form. This two-stage search-then-verify construction gives the framework its precision: embedding search p

Load-bearing premise

The framework assumes that a single multimodal-LLM description of an observation, after distillation and conversion, captures every task-relevant fact without omission, merging, or hallucination, and that the default nuisance filter (stripping color, size, and quantity) does not throw away information a particular task needs.

What would settle it

On a random sample of clips, have two independent human annotators list the atomic facts present, canonicalize them, and compare with the pipeline's bags: if the pipeline systematically omits facts (e.g., missing 'road is wet' on rainy frames) or hallucinates relations (e.g., labeling 'vehicle yielding' as 'vehicle stopping', a distinction the authors note embeddings blur), then retrieval precision and coverage-gap conclusions inherit those errors. A simpler targeted test: query for a concept the MLLM tends to merge with another and check whether BoAP retrieval separates the two behaviors.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Queries composed of several atomic propositions (e.g., 'pedestrian crossing street' AND 'red traffic light' AND 'road is wet') retrieve rare, long-tail scenes that a single free-form caption query cannot surface.
  • A corpus becomes a distribution over interpretable concepts, so training-test coverage gaps—such as crosswalk-yield behavior present in testing but absent in training—can be identified precisely enough to guide targeted data collection.
  • Because the same codebook is shared, semantically equivalent situations map to the same propositions regardless of source modality or dataset, enabling cross-dataset comparison.
  • The representation complements dense embeddings: it keeps perceptual robustness while adding an explicit, inspectable semantic layer for reasoning and curation.
  • The framework applies beyond autonomous driving to open-world video, suggesting the representation generalizes across domains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The codebook is built from a specific corpus, so an out-of-domain observation may contain propositions absent from the codebook; the paper says the codebook is incrementally extensible but does not specify an online update mechanism—an open design question for deployment.
  • Because proposition extraction depends on a single MLLM description call, systematic omissions or hallucinations in that model become systematic errors in every downstream retrieval and coverage statistic; ensembling or grounding against perception outputs would be a natural hardening step.
  • The coverage analysis could be turned into an active data-selection loop—automatically acquiring or synthesizing clips that contain rare proposition combinations—but the paper stops at identifying the gaps rather than closing them.
  • Retrieval here is exact set matching on canonical propositions; allowing weighted or fuzzy matching over the codebook could recover paraphrases not yet canonicalized, at the cost of some interpretability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes representing any multimodal observation (video, image, text, or sensor stream) as a bag of atomic propositions (APs): short declarative statements about entities, actions, attributes, and relations. A pipeline extracts APs from free-form MLLM descriptions via distillation, proposition conversion, and nuisance filtering (Sec. 2.1), then builds a global semantic codebook by clustering APs with embedding similarity and verifying equivalence with an LLM (Sec. 2.2, Algorithm 1). The claimed benefits are modality unification, compositional retrieval through set operations on APs, and interpretable corpus coverage analysis. Experiments on Physical AI, NavHard/NavTrain, and Sekai corpora include qualitative retrieval examples (Figs. 3–5), a small quantitative similarity comparison (Table 1), and descriptive coverage analyses (Figs. 6–8).

Significance. If the central claims were validated, the BoAP framework would be practically valuable for semantic indexing, long-tail data curation, and explainable retrieval, especially for autonomous driving. The paper is clearly written and includes explicit prompts and a formalized codebook construction algorithm (Algorithm 1). These are strengths: the pipeline is reproducible in structure, and the corpus scale (hundreds of thousands of clips) is ambitious. However, the current evidence is largely qualitative and self-referential. The only quantitative comparison is a four-video similarity table that does not measure retrieval performance, and the codebook is built and validated by the same model family that generates the APs, with no independent ground truth. The coverage analysis is descriptive. The central ideas are plausible and testable, but the evaluation as presented does not yet substantiate the paper's headline claims.

major comments (4)
  1. [§3.1, Table 1] The central comparative claim is not actually tested. Table 1 reports cosine similarities between queries and four hand-picked videos, but this is not a retrieval evaluation: there are no relevance judgments, no precision/recall@k, no error bars, and no dense-embedding baseline (e.g., a caption embedding or a composed-text embedding). The comparison is also skewed by construction: an AP query is decomposed into the same short strings that appear in the video's BoAP, so per-component similarity is near 1, whereas the caption baseline is a long string whose embedding dilutes each concept. Thus the table does not support the abstract's claim that BoAP enables compositional retrieval that dense embeddings or free-form captions do not support. A proper evaluation should use a pool of videos with human-labeled relevance, compare BoAP retrieval against dense retrieval and caption retrieval, and
  2. [§2.1.1–2.1.2, §2.2.3] The pipeline displays a circularity risk: Gemini-2.5-Pro is used both to extract APs and to validate semantic equivalence of codebook clusters, and Qwen3-Embedding-8B is used both for codebook grouping and for retrieval similarity. There is no human-annotated ground truth for the decomposition or the equivalence clusters. Accordingly, high retrieval similarity and clean cluster structure may reflect self-consistency of the model/embedder family rather than fidelity to human-judged scene semantics. Please add a human evaluation on random samples: (i) assess whether extracted APs are correct and complete relative to human annotation; (ii) assess whether cluster members are semantically equivalent to the leader. Report agreement rates. This is load-bearing because every downstream retrieval and coverage claim inherits codebook correctness.
  3. [§3.2, Figs. 6–8] The coverage analysis is descriptive. The paper asserts specific train/test gaps from t-SNE/KDE visualizations and bar charts without a defined gap metric, statistical test, or human validation. Since AP extraction and codebook errors propagate, visual impressions may be misleading. The authors should formalize coverage as a quantitative measure — for example, the fraction of NavHard test APs or AP combinations absent from NavTrain — and validate a sample of identified gaps by manual inspection. Without this, the 'knowledge gap' claims are not evidence-backed.
  4. [§2.1.1] The representation relies on the assumption that a single Gemini-2.5-Pro call per description and the subsequent conversion steps are lossless enough to capture all task-relevant semantics. This is asserted ('works well in practice') rather than measured. Omitted, merged, or hallucinated propositions will directly bias every retrieval and coverage result. Please quantify extraction faithfulness by comparing extracted APs to a human-annotated proposition set on a small sample of observations, and report precision/recall of AP extraction. This is necessary to support the central claim that BoAP preserves the semantic content of an observation.
minor comments (6)
  1. [Fig. 4] Caption contains a typo: 'infrascture' should be 'infrastructure'.
  2. [Table 1] 'AP Mean Combined Score' is not defined. Please state how the per-component scores are aggregated (e.g., arithmetic mean) and whether the caption score is comparable in scale.
  3. [§2.1.2, Eq. (1)] The formal AP definition as a tuple (r, a_1, ..., a_k) is not used in the examples, which are natural-language strings (Eq. 3). Clarify how the tuple notation relates to the string form used in extraction, retrieval, and the codebook.
  4. [Fig. 8] The caption text is run-on and contains a typo ('NavT rain'). Consider splitting into clearer sentences.
  5. [§2.1.3] The paper correctly notes that the nuisance modifier default set is configurable, but no sensitivity analysis is given. A short experiment showing how retrieval or coverage changes with different nuisance settings would strengthen the claims.
  6. [Algorithm 1] The candidate set N(l) is computed at line 9 but the algorithm later removes only {l} ∪ S. The text states that rejected candidates return to the pool; the algorithm can be made more explicit by showing that U ← U \ ({l} ∪ N(l)) ∪ (N(l) \ S), or by adding a comment that rejected candidates remain unassigned.

Circularity Check

2 steps flagged

Partial circularity: Table 1's AP-vs-caption retrieval comparison is a self-match artifact, and the codebook is validated in-loop by the same LLM/embedder that generates it.

specific steps
  1. self definitional [Sec. 3.1.1, Sec. 3.1.2, Sec. 3.2, Table 1]
    "An observation is retrieved when its bag of APs contains the queried concept. Retrieval is therefore matching over interpretable propositions ... whereas each AP keeps a significantly high similarity to its target and their combined score rises with complexity. This shows that composed APs are more effective for complex understanding than single free form caption."

    By construction, the videos in Table 1 are selected because their bags contain the queried canonical AP; querying with that same canonical string yields near-identity cosine scores. The caption condition embeds a different, longer string, so its lower score is an artifact of string length, not an independent measure of semantic retrieval quality. The reported superiority is therefore a restatement of the containment rule used to define BoAP retrieval, not an empirical validation of the central comparative claim.

  2. other [Sec. 2.2.3; Sec. 3 Models]
    "We use Gemini-2.5-Pro for all the processes during the semantic representation pipeline, specifically AP extraction and LLM semantic validation for codebook construction. For encoding AP text into embeddings, we utilize Qwen3-Embedding-8B for our leader-centric grouping and data retrieval experiments."

    The global semantic codebook is the load-bearing semantic ground truth for every downstream retrieval and coverage result, but it is constructed by letting the same model that generated the AP variants also judge which variants are semantically equivalent, and by using the same embedder to propose the clusters and later compute retrieval similarities. No human-annotated decomposition, relevance judgment, or external semantic benchmark enters the loop, so the reported high agreement can be read as self-consistency of Gemini/Qwen rather than fidelity to independently defined scene semantics.

full rationale

The framework itself has independent descriptive content: AP extraction, codebook clustering (Alg. 1), and the coverage statistics are real computations over the corpus, and the self-citations [20] and [22] are used only as prior baselines, not as load-bearing support. No uniqueness theorem or ansatz is imported from the authors. However, the paper's central comparative claim that BoAP enables compositional retrieval that dense/caption embeddings 'do not support' is not tested independently: Table 1 compares self-similarity of identical codebook strings against a long holistic caption embedding, so the outcome is fixed by the definition of AP retrieval rather than by relevance judgments or retrieval metrics. The codebook and retrieval loop also share the same LLM and embedder, so the semantic equivalence that underlies all downstream claims is only validated in-loop. This is partial circularity, not a fully forced derivation, so the score is 6 rather than 8 or 10.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 1 invented entities

The framework depends on LLM fidelity (extraction and verification), embedding quality, and the assumption that bag-of-propositions preserves task-relevant semantics. These are plausible but unvalidated against human ground truth. The main free parameters (tau, nuisance set, leader criteria) are heuristic choices without ablation or external calibration.

free parameters (3)
  • cosine similarity threshold tau
    Used in Algorithm 1 line 9 for leader-centric grouping; value not reported. Controls cluster granularity, codebook size, and retrieval behavior.
  • nuisance modifier default set = color, size, quantity
    Section 2.1.3: the set of modifiers removed as nuisance is a configurable design choice, acknowledged as task-specific.
  • codebook leader selection criteria = frequency -> length -> lexicographic
    Section 2.2.1: hierarchical ranking heuristic determines which AP becomes canonical; no ablation showing these criteria are optimal.
axioms (6)
  • domain assumption The MLLM description faithfully captures all salient semantic content of the observation.
    The entire extraction pipeline in Sec. 2.1 starts from a natural-language description generated by Gemini-2.5-Pro; no verification against ground-truth scene content.
  • domain assumption Distillation and proposition conversion via a single Gemini-2.5-Pro call are lossless with respect to task-relevant semantics.
    Sec. 2.1.1-2.1.2 rely on LLM decomposition into single-clause declaratives and triplets; errors propagate to all downstream uses.
  • domain assumption Embedding cosine similarity above tau is a valid proxy for semantic relatedness (Qwen3-Embedding-8B).
    Sec. 2.2.2 uses embedding similarity to propose candidate clusters; if embeddings miss semantic paraphrases, the codebook is incomplete.
  • domain assumption LLM semantic-equivalence verification yields correct equivalence judgments.
    Sec. 2.2.3: the same LLM family defines which propositions are equivalent; no human-annotated equivalence benchmark is provided.
  • ad hoc to paper A bag-of-propositions representation (order irrelevant, limited quantifier handling, retained negations) is sufficient to support the claimed reasoning and retrieval.
    Sec. 2.1.2 defines APs as triplets; the adequacy of this simplification for complex semantics (e.g., negation scope, counting, temporal relations) is asserted rather than demonstrated.
  • domain assumption Nuisance filtering removes only task-irrelevant modifiers.
    Sec. 2.1.3 explicitly states what counts as nuisance is task-specific; the default filter may discard information needed for some tasks.
invented entities (1)
  • Atomic proposition (AP) as a discrete semantic token no independent evidence
    purpose: Fundamental unit of the proposed representation; each AP is a small declarative statement used to represent observations and queries.
    APs are a representational reification of SVO triplets; no human-annotated ground truth shows that scenes decompose cleanly into such atoms. Their validity is inferred from LLM consensus within the pipeline.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of From Modalities to Propositions: A Language-Centric Framework for Multimodal Intelligence." pith.science (2026). https://pith.science/paper/2UCQLK35

@misc{pith2026260716560,
  author       = {Pith},
  title        = {Pith review of: From Modalities to Propositions: A Language-Centric Framework for Multimodal Intelligence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2UCQLK35}},
  note         = {Machine review of arXiv:2607.16560}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We propose a language representation for multimodal data in which any observation, whether image, video, or text, is expressed as a bag of atomic propositions, simple statements about the entities, actions, and relations in a scene. A global semantic codebook unifies these into a shared vocabulary of canonical atomic propositions, placing every modality and observation into one interpretable space that spans fine grained facts to high level concepts and composes into richer ones. This brings interpretability with reasoning, cross-modal understanding and retrieval, and compositionality that enables complex multimodal understanding, rich data curation and complex structured retrieval. We demonstrate the framework on autonomous driving and open-world data.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 6 linked inside Pith

  1. [1]

    Qwen-vl: A versatile vision-language model for understand- ing, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understand- ing, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Mech- anistic interpretability for ai safety–a review

    Leonard Bereska and Efstratios Gavves. Mech- anistic interpretability for ai safety–a review. arXiv preprint arXiv:2404.14082, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    On the oppor- tunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the oppor- tunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

    Pith/arXiv arXiv 2021

  4. [4]

    Naver: A neuro-symbolic compositional automaton for visual grounding with explicit logic reasoning

    Zhixi Cai, Fucai Ke, Simindokht Jahangard, Maria Garcia de la Banda, Reza Haffari, Pe- ter J Stuckey, and Hamid Rezatofighi. Naver: A neuro-symbolic compositional automaton for visual grounding with explicit logic reasoning. In ICCV, pages 24078–24089, 2025

    2025

  5. [5]

    Pseudo-simulation for autonomous driving.arXiv preprint arXiv:2506.04218, 2025

    Wei Cao, Marcel Hallgarten, Tianyu Li, Daniel Dauner, Xunjiang Gu, Caojun Wang, Yakov Miron, Marco Aiello, Hongyang Li, Igor Gilitschenski, et al. Pseudo-simulation for autonomous driving.arXiv preprint arXiv:2506.04218, 2025

    arXiv 2025

  6. [6]

    Position: Stop reactively patching your model every time and start proactive test-driven ai development

    Nadine Chang, Maying Shen, Jialiang Wang, Rafid Mahmood, and Jose M Alvarez. Position: Stop reactively patching your model every time and start proactive test-driven ai development. InICML, 2026. 10 From Modalities to Propositions: A Language-Centric Framework for Multimodal Intelligence

    2026

  7. [7]

    M3- embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self- knowledge distillation

    Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3- embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self- knowledge distillation. InFindings of the asso- ciation for computational linguistics: ACL 2024, pages 2318–2335, 2024

    2024

  8. [8]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodal- ity, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaek- ermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodal- ity, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    NAVSIM: Data-driven non-reactive au- tonomous vehicle simulation and benchmarking

    Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, and Kashyap Chitta. NAVSIM: Data-driven non-reactive au- tonomous vehicle simulation and benchmarking. InNeurIPS, volume 37, 2024

    2024

  10. [10]

    Toy models of superposition.arXiv preprint arXiv:2209.10652, 2022

    Nelson Elhage, Tristan Hume, Catherine Ols- son, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition.arXiv preprint arXiv:2209.10652, 2022

    Pith/arXiv arXiv 2022

  11. [11]

    Martin.Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition

    Dan Jurafsky and James H. Martin.Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice Hall, Upper Saddle River, NJ, 2nd edition, 2009

    2009

  12. [12]

    Explainability and vision foundation models: A survey.Information Fusion, 122:103184, 2025

    Rémi Kazmierczak, Eloïse Berthier, Goran Frehse, and Gianni Franchi. Explainability and vision foundation models: A survey.Information Fusion, 122:103184, 2025

    2025

  13. [13]

    Sekai: A video dataset towardsworldexploration

    Zhen Li, Chuanhao Li, Xiaofeng Mao, Shaoheng Lin, Ming Li, Shitian Zhao, Zhaopan Xu, Xinyue Li, Yukang Feng, Jianwen Sun, Zizhen Li, Fanrui Zhang, Jiaxin Ai, Zhixiang Wang, Yuwei Wu, Tong He, Jiangmiao Pang, Yu Qiao, Yunde Jia, and Kaipeng Zhang. Sekai: A video dataset towardsworldexploration. InAdvances in Neural Information Processing Systems, volume 38...

    2025

  14. [14]

    A comprehensive survey and guide to multimodal large language models in vision–language tasks

    Chia Xin Liang, Pu Tian, Caitlyn Heqi Yin, Yao Yua, An-Hou Wei, Ming Li, Xinyuan Song, Tianyang Wang, Ziqian Bi, Ming Liu, et al. A comprehensive survey and guide to multimodal large language models in vision–language tasks. Computation, 14(6):125, 2026

    2026

  15. [15]

    Visual instruction tuning.Ad- vances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Ad- vances in neural information processing systems, 36:34892–34916, 2023

    2023

  16. [16]

    Manning and Hinrich Schütze

    Christopher D. Manning and Hinrich Schütze. Foundations of Statistical Natural Language Pro- cessing. MIT Press, Cambridge, MA, 1999

    1999

  17. [17]

    Enhancing reason- ing capabilities of llms via principled synthetic logic corpus.Advances in Neural Information Processing Systems, 37:73572–73604, 2024

    Terufumi Morishita, Gaku Morio, Atsuki Yam- aguchi, and Yasuhiro Sogawa. Enhancing reason- ing capabilities of llms via principled synthetic logic corpus.Advances in Neural Information Processing Systems, 37:73572–73604, 2024

    2024

  18. [18]

    PhysicalAI–Autonomous Vehicles

    NVIDIA Corporation. PhysicalAI–Autonomous Vehicles. https://huggingface.co/datasets/ nvidia/PhysicalAI-Autonomous-Vehicles,

  19. [19]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  20. [20]

    Sse: Multimodal semantic data selection and enrichment for industrial-scale data assimilation

    Maying Shen, Nadine Chang, Sifei Liu, and Jose M Alvarez. Sse: Multimodal semantic data selection and enrichment for industrial-scale data assimilation. InKDD, pages 2525–2535, 2025

    2025

  21. [21]

    Winoground: Probing vision and language models for visio-linguistic composition- ality

    TristanThrush, RyanJiang, MaxBartolo, Aman- preet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic composition- ality. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recogni- tion, pages 5238–5248, 2022

    2022

  22. [22]

    Scalable parallel prompting for complex av video captioning

    April Yang, Roberto Amoroso, Nikita Durasov, Devansh Bisla, Sandipan Kundu, Elmar Hauss- mann, Ruchi Bhargava, Maying Shen, Nadine Chang, and Jose M Alvarez. Scalable parallel prompting for complex av video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4407–4416, 2026

    2026

  23. [23]

    there"or

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176, 2025. 11 From Modalities to Propositions: A Language-Centric Framework for Multimodal Intelligence A. Prompts ...

    Pith/arXiv arXiv 2025

  24. [2026]

    Accessed July 16, 2026

    Dataset card, version 26.03. Accessed July 16, 2026

    2026

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.