Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

IndustryEQA: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarios

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces IndustryEQA, the first benchmark that tests embodied agents' question answering inside safety-critical warehouse scenarios, with 1,344 QA pairs over 76 simulated egocentric videos and baselines showing visual input…

desk verdict Useful first warehouse EQA benchmark, but ground-truth reliability is unproven and the appendix shows concrete errors, so treat leaderboard scores as provisional. read the letter →

arxiv 2505.20640 v1 pith:UKZ23J4Y submitted 2025-05-27 cs.CV

Yifan Li , Yuhang Chen , Anh Dao , Lichi Li , Zhongyi Cai , Zhen Tan , Tianlong Chen , Yu Kong This is my paper · ORCID
classification cs.CV
keywords embodiedquestionansweringindustrialwarehousesafetybenchmarkvision-languagemodelsepisodicmemorysimulationsafety-criticalAIevaluationvideo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

IndustryEQA is the first benchmark purpose-built to test how well embodied agents understand safety-critical industrial environments through natural-language questions. The paper builds 76 high-fidelity warehouse scenes in the Isaac Sim simulator, records egocentric navigation videos from them, and releases 1,344 human-filtered question–answer pairs spanning six categories — equipment safety, human safety, object recognition, attribute recognition, temporal understanding, and spatial understanding — with a subset carrying explicit reasoning answers. A baseline evaluation of blind text-only models, multi-frame vision-language models, and video models establishes that visual grounding is essential and that reasoning about safety remains the hardest part, with temporal understanding the weakest category and a uniform drop from direct to reasoning scores. If this holds, safety-aware industrial evaluation becomes a measurable axis of embodied AI progress rather than an assumed by-product of general benchmarks.

What carries the argument

The load-bearing object is the benchmark construction itself: simulated hazardous warehouse scenes assembled from the Isaac Sim asset library, populated according to real safety-guideline scenarios, with each layout converted into an egocentric episodic-memory video. The QA pipeline that carries the evaluation is a three-stage chain — an initial generation of question–answer pairs by a vision-language model guided by a safety-focused prompt (at least half safety-related), a transformation stage that rewrites eligible yes/no items into open-ended or multiple-choice questions, and an LLM-based refinement stage followed by manual expert filtering and reannotation. Scoring comes from the LLM-Match protocol, in which a judge model rates each agent answer against the ground truth on a 1–5 scale normalized by $$C = \frac{1}{N}\sum_{i=1}^N \frac{\sigma_i - 1}{4} \times 100\%$$, yielding separate Direct and Reasoning scores. Baselines are split into blind text-only models, multi-frame VLLMs, and video VLLMs, which is what makes the paper's central contrasts — vision versus no vision, and direct recall versus reasoning — measurable.

What would settle it

Have independent expert annotators re-answer a random sample of the 1,344 questions from the videos and measure agreement with the released ground truth and with each other; if agreement on the safety and reasoning answers is low, the leaderboard scores cannot support conclusions about model capabilities. A lighter, immediate check is to audit the released JSON for the failure patterns already visible — duplicated question IDs (705 and 709), a reasoning answer for question 658 that describes aisle layout rather than the overhead utilities asked about, and "Causal Reasoning" labels (757, 793) that do not appear in the declared six-category taxonomy.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that an industrial, safety-focused EQA benchmark can be built entirely from simulation and still separate model capabilities in ways household benchmarks do not. Using Isaac Sim, the authors create small and large warehouse layouts with deliberately hazardous configurations derived from U.S. Department of Labor warehouse safety guidance — unsecured ladders, tipping containers, blocked fire-fighting equipment, unprotected pedestrian–forklift lane sharing — and with dynamic human workers, an element previous EQA benchmarks omit. Episodic-memory videos of a camera-equipped Carter robot navigating each scene are paired with machine-generated, expert-filtered QA annotations. Evaluating blind LLMs, multi-frame VLLMs, and video VLLMs in a zero-shot protocol with LLM-judge scoring, the paper reports that the best visual models exceed 65% direct score while blind models cluster near 40%, that reasoning scores trail direct scores across nearly all models and categories, and that temporal understanding is the hardest category while attribute recognition is easiest.

Load-bearing premise

Everything hangs on the assumption that the human-filtered question–answer pairs are correct, unambiguous, and answerable from the video alone, and the paper reports no measurement of annotation reliability, such as inter-annotator agreement or error rates, while the released examples include duplicated questions, a reasoning answer that describes the wrong evidence, and category labels outside the declared six types.

Editorial extensions

If this is right

  • If IndustryEQA is right that safety reasoning separates models even when direct answers match, safety-aware QA should become a standard reporting axis in embodied AI evaluations, and safety deficits should be treated as capability gaps rather than noise.
  • The benchmark gives embodied AI a concrete deployment-readiness test: an agent that cannot answer these 1,344 questions at near-perfect reasoning score is not ready to operate unsupervised in a real warehouse.
  • Because the scenes are generated procedurally, the benchmark can be extended cheaply with new hazard types, more extreme events, and audio tracks, all listed as future work, allowing the community to track whether safety reasoning improves as the environment gets harder.
  • The frame-density ablation implies that evaluations on long egocentric videos need dense temporal sampling before conclusions about temporal understanding are trustworthy, since scores rise with more sampled frames across all tested models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper leaves implicit: the same Isaac Sim scenes could support an active-navigation variant in which the agent must physically explore to locate hazards, converting the benchmark from episodic-memory grading into a training environment for navigation and reinforcement learning.
  • The direct-versus-reasoning score gap could be converted into a training signal: the reasoning annotations could ground a reward model that fine-tunes vision-language or policy models specifically for safety justification, which the paper mentions only as a possibility.
  • The blind-model scores near 40% quantify how much of industrial QA can be answered from warehouse world-knowledge alone, so the vision-vs-blind gap is itself a measurable estimate of how much of the task is genuinely visual — a quantity worth reporting per category.
  • The six-category difficulty ordering (attribute recognition easiest, temporal hardest, safety in the middle) is plausibly transferable to other safety-critical domains such as construction sites or hospitals, where the same video-plus-QA pipeline could be reapplied.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces IndustryEQA, a simulated warehouse EQA benchmark built in NVIDIA Isaac Sim, with 76 egocentric videos (60 small and 16 large warehouse layouts) and 1,344 QA pairs covering six named categories (equipment safety, human safety, object recognition, attribute recognition, temporal understanding, spatial understanding) plus an additional reasoning-answer subset. The authors describe a pipeline of VLLM-based QA generation, transformation, LLM-based refinement, and human filtering, and they propose an LLM-judge evaluation protocol with Direct and Reasoning scores. They evaluate blind LLMs, multi-frame VLLMs, and video VLLMs, and report findings that visual grounding is critical, reasoning is harder than direct answering, and safety-related questions remain challenging. The benchmark and code are publicly released.

Significance. If the annotations are reliable, IndustryEQA fills a genuine gap: no prior EQA benchmark targets safety-critical warehouse scenarios with human agents, episodic video, and reasoning answers. The controlled Isaac Sim data collection with OSHA-inspired hazards, the public release, the LLM-judge protocol, and the comparison across blind, multi-frame, and video models are concrete strengths that make the benchmark potentially useful to the embodied-AI and VQA communities. The central value of the paper is the gold annotations, however, and the paper's own appendix exposes failures in those gold annotations; until the dataset is audited and corrected, the leaderboard numbers and the conclusions drawn from them should be treated as provisional.

major comments (4)
  1. [Appendix C.1 and Fig. 11] The stated six-category taxonomy is not the one actually used in the released annotations. The generation prompt in Appendix C.1 instructs the model to 'explicitly generate some difficult Causal Reasoning and Commonsense Reasoning type questions,' and Fig. 11 shows QA pairs 757 and 793 with type 'Causal Reasoning,' a type absent from the six categories defined in Sections 3.2 and 3.3 and absent from the distribution in Fig. 4. Because the category-wise scores in Figs. 6 and 12 are computed from type labels, the benchmark's category statistics and the claim that each QA pair was validated against one of six predefined categories are not supported. The authors should either fold these labels into an explicit taxonomy or remove them and re-release corrected category statistics.
  2. [Fig. 11, QA 658] The ground-truth reasoning answer does not address the question. The question asks about overhead utilities and the direct answer is '(B) Some electrical cables are exposed and hanging,' but the reasoning answer describes aisle racking and stacked pallets. Under the reasoning-score protocol of Eq. (1), a model that supplies the correct causal evidence for the exposed cables would be scored as mismatching the gold reasoning, while a model that repeats the aisle statement would be rewarded. This is a concrete ground-truth error that must be corrected or the QA must be removed from the benchmark.
  3. [Fig. 11, IDs 705 and 709] The same question, direct answer, and reasoning answer appear twice under different type labels ('Attribute Recognition' and 'Equipment Safety'), indicating both a deduplication failure and a category-validation failure. Duplicated entries inflate category counts and can bias aggregate and category-wise scores; the released dataset needs a duplicate check and a report of how many duplicates were found and removed.
  4. [Section 3.2, Human Filtering and Reannotation] The manuscript claims that 'rigorous human oversight ensures a high-quality benchmark,' but it provides no quantitative reliability evidence such as inter-annotator agreement, counts of QAs removed or edited per stage, or a post-release audit. Given the counterexamples in the appendix, this is not a cosmetic concern: the gold answers are the basis for every Direct and Reasoning Score in Table 2, so the absence of reliability statistics, together with visible errors, undermines the benchmark's central claim until an audit is supplied.
minor comments (6)
  1. [Section 3.3] 'emobodied agents' should be 'embodied agents'.
  2. [Fig. 11 caption] 'QA paris' should be 'QA pairs'.
  3. [Table 1] The header lists seven Boolean capability columns (Safety, Reasoning, Open Vocab, Human, LLM scoring, EM, Video), but each data row contains only six Boolean symbols; the status of the Video column is unresolved.
  4. [Section 4.3, finding 2] The text refers to 'complex causal, spatial, and temporal understanding,' but 'causal' is not one of the six annotation categories; the finding should be aligned with the actual taxonomy.
  5. [Section 3.2 and Appendix C.1] Section 3.2 says the initial generation phase produced 'over 3,000 QA pairs,' while Appendix C.1 says 'over 2,000 QA pairs'; this discrepancy should be reconciled.
  6. [Fig. 4] The caption mentions 'red dimonds' (sic), and the inner/outer ring encoding is not described in the caption; please clarify the figure legend.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: IndustryEQA is a self-contained dataset and evaluation contribution; its empirical findings are not derived from fitted parameters or self-citational assumptions.

full rationale

I traced the paper's load-bearing chain: simulated warehouse video creation, VLLM-based QA generation, human expert filtering, fixed ground-truth answers, LLM-based scoring, and the empirical comparisons. Nothing in this chain defines a reported result in terms of its own inputs and then presents that definition as a prediction. The scoring formula in Eq. (1) is only a linear normalization of an external judge's 1-5 ratings, and the judge is a fixed, pre-existing LLM whose sensitivity is explicitly ablated in Fig. 8. The paper's one self-citation (ref. [19], the authors' VLLM survey) merely motivates the episodic-memory setting and is not load-bearing; no uniqueness theorem or ansatz is imported from the authors' prior work. The appendix data-quality artifacts (duplicate IDs 705/709, the mismatched reasoning answer for ID 658, and out-of-taxonomy 'Causal Reasoning' labels) are genuine construct-validity threats, but they are not circular reductions of the benchmark conclusions to the benchmark's inputs. The simulated hazards are grounded in an external OSHA source, and the baseline comparisons are standard external models. Thus the paper's central claims are self-contained with respect to circularity, and no circular step is exhibited.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The benchmark's validity rests on assumptions about simulation fidelity, annotation reliability, and LLM judge behavior. No free parameters are fitted to data in the physics sense; the only hand-chosen numeric is the frame sampling count K.

free parameters (1)
  • Number of sampled frames K for multi-frame VLLMs = 30 (small warehouse), 40 (large warehouse)
    Hand-chosen default for the multi-frame VLLM baseline; the ablation study shows scores improve with K, but no principled criterion is given for the default.
assumptions (4)
  • domain assumption Isaac Sim provides sufficiently realistic warehouse scenes and physics for embodied EQA evaluation.
    The paper uses Isaac Sim exclusively and does not validate against real warehouse videos.
  • domain assumption The OSHA 'Hazards and Solutions' documentation is a valid and sufficient basis for designing safety-critical scenarios.
    The paper draws hazard types from this guidance without verifying that the simulated hazards are perceived as intended.
  • domain assumption LLM-generated QA pairs, after expert filtering, constitute correct and unambiguous ground truth.
    The data pipeline relies on Gemini 2.5 Pro for initial QA generation and human filtering for correction, but no annotation quality metrics are reported.
  • domain assumption An LLM judge can reliably score open-ended answers against the ground truth.
    The evaluation uses GPT-4o-mini as judge; the judge sensitivity ablation shows variability, so this assumption is only partially tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IndustryEQA: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarios." pith.science (2026). https://pith.science/paper/UKZ23J4Y

@misc{pith2026250520640,
  author       = {Pith},
  title        = {Pith review of: IndustryEQA: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarios},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UKZ23J4Y}},
  note         = {Machine review of arXiv:2505.20640}
}
read the original abstract

Existing Embodied Question Answering (EQA) benchmarks primarily focus on household environments, often overlooking safety-critical aspects and reasoning processes pertinent to industrial settings. This drawback limits the evaluation of agent readiness for real-world industrial applications. To bridge this, we introduce IndustryEQA, the first benchmark dedicated to evaluating embodied agent capabilities within safety-critical warehouse scenarios. Built upon the NVIDIA Isaac Sim platform, IndustryEQA provides high-fidelity episodic memory videos featuring diverse industrial assets, dynamic human agents, and carefully designed hazardous situations inspired by real-world safety guidelines. The benchmark includes rich annotations covering six categories: equipment safety, human safety, object recognition, attribute recognition, temporal understanding, and spatial understanding. Besides, it also provides extra reasoning evaluation based on these categories. Specifically, it comprises 971 question-answer pairs generated from small warehouse and 373 pairs from large ones, incorporating scenarios with and without human. We further propose a comprehensive evaluation framework, including various baseline models, to assess their general perception and reasoning abilities in industrial environments. IndustryEQA aims to steer EQA research towards developing more robust, safety-aware, and practically applicable embodied agents for complex industrial environments. Benchmark and codes are available.

Figures

Figures reproduced from arXiv: 2505.20640 by the authors.

Figure 1
Figure 1. An illustration of the IndustryEQA benchmark, consisting of episodic memory videos [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An illustration of industrial scenarios in two [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An illustration of data generation pipeline. It consists of three main steps, capturing the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: IndustryEQA statistics for small and large warehouses: question category distribution (pie [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: An illustration of the question ID 364 in small warehouse. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Category-wise performance comparison on the IndustryEQA small-warehouse scenario. (a) [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Impact of different sampled frame density w.r.t. (a) [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Sensitivity to LLM judge w.r.t. (a) Direct Score and (b) [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Distribution of the top 50 word frequencies in the small-warehouse QA data (from top to [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Distribution of the top 50 word frequencies in the large-warehouse QA data (from top to [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Examples of IndustryEQA benchmark QA paris. [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Category-wise performance comparison on the IndustryEQA large-warehouse scenario. [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Impact of different sampled frame density w.r.t. (a) Direct Score and (b) Reasoning Score [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: New version under development. D.2 Evaluation Protocol Model responses were assessed using Direct Scores and Reasoning Scores, calculated via an LLM judge (GPT-4o-mini and Gemini-2.0-flash). A 1-5 scale was used, with scores normalized to a percentage as shown in the …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigation

    cs.RO 2025-11 conditional novelty 5.0 of 10

    On IndustryNav, a dynamic Unity warehouse navigation benchmark, nine VLLMs earned only 4.9–65.3% success and high collision/warning rates, with closed-source models ahead.

  2. The high-speed X-ray camera on AXIS: design and performance updates

    astro-ph.IM 2025-08 unverdicted novelty 4.0 of 10

    An X-ray camera design-update whose supporting full text is a different paper (RynnEC, an embodied AI model), leaving all camera performance claims unverified.

Reference graph

Works this paper leans on

50 extracted references · 44 canonical work pages · cited by 2 Pith papers

  1. [1]

    Model card addendum: Claude 3.5 haiku and upgraded claude 3.5 sonnet

    Anthropic. Model card addendum: Claude 3.5 haiku and upgraded claude 3.5 sonnet. Technical report, Anthropic PBC, Oct. 2024. 8

    work page 2024

  2. [2]

    Azuma, T

    D. Azuma, T. Miyanishi, S. Kurita, and M. Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In CVPR, pages 19129–19139, 2022. 2, 3

    work page 2022

  3. [3]

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 8

    arXiv 2025

  4. [4]

    J. Chen, R. Huang, Z. Lv, J. Tang, and W. Li. Faultgpt: Industrial fault diagnosis question answering system by vision language models. arXiv preprint arXiv:2502.15481, 2025. 3

    arXiv 2025

  5. [5]

    Z. Chen, W. Wang, Y . Cao, Y . Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liu, L. Gu, X. Wang, Q. Li, Y . Ren, Z. Chen, J. Luo, J. Wang, T. Jiang, B. Wang, C. He, B. Shi, X. Zhang, H. Lv, Y . Wang, W. Shao, P. Chu, Z. Tu, T. He, Z. Wu, H. Deng, J. Ge, K. Chen, K. Zhang, L. Wang, M. Dou, L. Lu, X. Zhu, T. Lu, D. Lin, Y . Qiao, J. Dai, and W. Wang. Exp...

    work page 2025

  6. [6]

    Cobbe, V

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021. 17

    work page 2021

  7. [7]

    A. Das, S. Datta, G. Gkioxari, S. Lee, D. Parikh, and D. Batra. Embodied question answering. In CVPR, pages 1–10, 2018. 1, 2, 3

    work page 2018

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. 8

    work page 2025

Show all 50 references
  1. [9]

    Deepseek-v3 technical report, 2025

    DeepSeek-AI. Deepseek-v3 technical report, 2025. 8

    2025

  2. [10]

    Introducing gemini 2.0 flash: Our fastest multimodal model yet, December 2024

    Google DeepMind. Introducing gemini 2.0 flash: Our fastest multimodal model yet, December 2024. 8

    2024

  3. [11]

    Gemini 2.5 Flash Preview Model Card

    Google DeepMind. Gemini 2.5 Flash Preview Model Card. Technical report, Mountain View, CA, april

  4. [12]

    Gemini 2.5 Pro Preview Model Card

    Google DeepMind. Gemini 2.5 Pro Preview Model Card. Technical report, Mountain View, CA, may 2025. 4, 5

    2025

  5. [13]

    Gordon, A

    D. Gordon, A. Kembhavi, M. Rastegari, J. Redmon, D. Fox, and A. Farhadi. Iqa: Visual question answering in interactive environments. In CVPR, pages 4089–4098, 2018. 2

    2018

  6. [14]

    Khanov, J

    M. Khanov, J. Burapacheep, and Y . Li. ARGS: Alignment as reward-guided search. In The Twelfth International Conference on Learning Representations, 2024. 17

    2024

  7. [15]

    Kolve, R

    E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y . Zhu, A. Kembhavi, A. Gupta, and A. Farhadi. Ai2-thor: An interactive 3d environment for visual ai,

  8. [16]

    H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. Lu, C. Bishop, E. Hall, V . Carbune, A. Rastogi, and S. Prakash. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback,

  9. [17]

    G. Li, H. Du, and D. Hu. Boosting audio visual question answering via key semantic-aware cues, 2024. 17

    2024

  10. [18]

    W. Li, X. Zhang, S. Zhao, Y . Zhang, J. Li, L. Zhang, and J. Zhang. Q-insight: Understanding image quality via visual reinforcement learning, 2025. 17

    2025

  11. [19]

    Y . Li, Z. Lai, W. Bao, Z. Tan, A. Dao, K. Sui, J. Shen, D. Liu, H. Liu, and Y . Kong. Visual large language models for generalized and specialized applications. arXiv preprint arXiv:2501.02765, 2025. 3

    2025 arXiv

  12. [20]

    Lightman, V

    H. Lightman, V . Kosaraju, Y . Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step, 2023. 17

    2023

  13. [21]

    L. Luo, Y . Liu, R. Liu, S. Phatale, M. Guo, H. Lara, Y . Li, L. Shu, Y . Zhu, L. Meng, J. Sun, and A. Rastogi. Improve mathematical reasoning in language models by automated process supervision, 2024. 17

    2024

  14. [22]

    X. Ma, S. Yong, Z. Zheng, Q. Li, Y . Liang, S.-C. Zhu, and S. Huang. Sqa3d: Situated question answering in 3d scenes. In ICLR, 2023. 2 26

    2023

  15. [23]

    Majumdar, A

    A. Majumdar, A. Ajay, X. Zhang, P. Putta, S. Yenamandra, M. Henaff, S. Silwal, P. Mcvay, O. Maksymets, S. Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. In CVPR, pages 16488–16498, 2024. 1, 2, 3, 6

    2024

  16. [24]

    Makoviychuk, L

    V . Makoviychuk, L. Wawrzyniak, Y . Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance gpu-based physics simulation for robot learning,

  17. [25]

    Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, Y . Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V . Koltun, J. Malik, D. Parikh, and D. Batra. Habitat: A Platform for Embodied AI Research. In ICCV, 2019. 2

    2019

  18. [26]

    Manuvinakurike, E

    R. Manuvinakurike, E. Watkins, C. Savur, A. Rhodes, S. Biswas, G. G. Mejia, R. Beckwith, S. Sahay, G. Raffa, and L. Nachman. Qa-toolbox: Conversational question-answering for process task guidance in manufacturing. arXiv preprint arXiv:2412.02638, 2024. 3

    2024 arXiv

  19. [27]

    Llama 4 model card, Apr

    Meta AI. Llama 4 model card, Apr. 2025. 8

    2025

  20. [28]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. 8, 9

    2024

  21. [29]

    Introducing gpt-4.1 in the api, Apr

    OpenAI. Introducing gpt-4.1 in the api, Apr. 2025. 8

    2025

  22. [30]

    Openai o3 and o4-mini system card

    OpenAI. Openai o3 and o4-mini system card. Technical report, Apr. 2025. 5, 8

    2025

  23. [31]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions ...

    2022

  24. [32]

    X. Puig, E. Undersander, A. Szot, M. D. Cote, R. Partsey, J. Yang, R. Desai, A. W. Clegg, M. Hlavac, T. Min, T. Gervet, V . V ondrus, V .-P. Berges, J. Turner, O. Maksymets, Z. Kira, M. Kalakrishnan, J. Malik, D. S. Chaplot, U. Jain, D. Batra, A. Rai, and R. Mottaghi. Habitat ...

    2023

  25. [33]

    Rafailov, A

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing...

    2023

  26. [34]

    A. C. Rivera, A. Moore, and S. Robinson. Coal mining question answering with llms. arXiv preprint arXiv:2410.02959, 2024. 3

    2024 arXiv

  27. [35]

    Sener, D

    F. Sener, D. Chatterjee, D. Shelepov, K. He, D. Singhania, R. Wang, and A. Yao. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In CVPR, pages 21096–21106,

  28. [36]

    Sermanet, T

    P. Sermanet, T. Ding, J. Zhao, F. Xia, D. Dwibedi, K. Gopalakrishnan, C. Chan, G. Dulac-Arnold, S. Maddineni, N. J. Joshi, et al. Robovqa: Multimodal long-horizon reasoning for robotics. pages 645–652,

  29. [37]

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . K. Li, Y . Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. 17

    2024

  30. [38]

    A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y . Zhao, J. Turner, N. Maestre, M. Mukadam, D. Chaplot, O. Maksymets, A. Gokaslan, V . V ondrus, S. Dharur, F. Meier, W. Galuba, A. Chang, Z. Kira, V . Koltun, J. Malik, M. Savva, and D. Batra. Habitat 2.0: Training home assistan...

    2021

  31. [39]

    S. Tan, M. Ge, D. Guo, H. Liu, and F. Sun. Knowledge-based embodied question answering. IEEE TPAMI, 45(10):11948–11960, 2023. 2

    2023

  32. [40]

    P. Wang, L. Li, Z. Shao, R. X. Xu, D. Dai, Y . Li, D. Chen, Y . Wu, and Z. Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations, 2024. 17

    2024

  33. [41]

    Y . Wang, Z. Li, Y . Zang, C. Wang, Q. Lu, C. Jin, and J. Wang. Unified multimodal chain-of-thought reward model through reinforcement fine-tuning, 2025. 17 27

    2025

  34. [42]

    Y . Wang, Z. Tan, J. Wang, X. Yang, C. Jin, and H. Li. Lift: Leveraging human feedback for text-to-video model alignment, 2025. 17

    2025

  35. [43]

    Y . Wang, Y . Zang, H. Li, C. Jin, and J. Wang. Unified reward model for multimodal understanding and generation, 2025. 17

    2025

  36. [44]

    Wijmans, S

    E. Wijmans, S. Datta, O. Maksymets, A. Das, G. Gkioxari, S. Lee, I. Essa, D. Parikh, and D. Batra. Embodied question answering in photorealistic environments with point cloud perception. In CVPR, pages 6659–6668, 2019. 2

    2019

  37. [45]

    Y . Wu, Y . Wu, G. Gkioxari, and Y . Tian. Building generalizable agents with a realistic and rich 3d environment. arXiv preprint arXiv:1801.02209, 2018. 2

    2018 arXiv

  38. [46]

    Xiong, X

    T. Xiong, X. Wang, D. Guo, Q. Ye, H. Fan, Q. Gu, H. Huang, and C. Li. Llava-critic: Learning to evaluate multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2025. 17

    2025

  39. [47]

    F. Yang, P. Zhao, Z. Wang, L. Wang, J. Zhang, M. Garg, Q. Lin, S. Rajmohan, and D. Zhang. Empower large language model to perform better on industrial domain-specific question answering. arXiv preprint arXiv:2305.11541, 2023. 3

    2023 arXiv

  40. [48]

    P. Yang, X. Wang, X. Duan, H. Chen, R. Hou, C. Jin, and W. Zhu. Avqa: A dataset for audio-visual question answering on videos. In Proceedings of the 30th ACM International Conference on Multimedia, MM ’22, page 3480–3491, New York, NY , USA, 2022. Association for Computing Mac...

    2022

  41. [49]

    L. Yu, X. Chen, G. Gkioxari, M. Bansal, T. L. Berg, and D. Batra. Multi-target embodied question answering. In CVPR, pages 6309–6318, 2019. 2

    2019

  42. [50]

    Y . Zhao, K. Xu, Z. Zhu, Y . Hu, Z. Zheng, Y . Chen, Y . Ji, C. Gao, Y . Li, and J. Huang. Cityeqa: A hierarchical llm agent on embodied question answering benchmark in city space. arXiv preprint arXiv:2502.12532,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.