Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

LLaVA-Pose: Enhancing Human Pose and Action Understanding via Keypoint-Integrated Instruction Tuning

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Human keypoint annotations, when folded into instruction-following data, lift a vision-language model's pose and action understanding by 33.2 percent on a new benchmark.

desk verdict Useful recipe and dataset, but the headline 33.2% gain rests on a self-referential GPT-4o judge on a self-built benchmark with no keypoint-free control, so treat the quantitative claim as unproven. read the letter →

arxiv 2506.21317 v1 pith:JIEI74CR submitted 2025-06-26 cs.CV

Dewen Zhang , Tahir Hussain , Wangpeng An , Hayaru Shouno This is my paper · ORCID
classification cs.CV
keywords humanposeandactionunderstandingkeypoint-integrateddatagenerationinstruction-followingvision-languagemodelsmultimodalinstructiontuningLLaVA-PoseE-HPAUBbenchmarkGPT-4o
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the weak human pose and action performance of vision-language models is fundamentally a data problem: their instruction-following data describes objects and scenes, not bodies. The proposed fix is to feed a language-only teacher model (GPT-4o) a symbolic context made of captions, object bounding boxes, and per-person keypoints, the 17 joint locations used in COCO, and to have it generate instruction samples in three forms: conversation, detailed description, and complex reasoning. Fine-tuning LLaVA-1.5-7B on the resulting 200,328 samples produces LLaVA-Pose, which scores 69.4 on the new Extended Human Pose and Action Understanding Benchmark (E-HPAUB) versus 52.1 for the original model, a 33.2% relative improvement. The paper also claims this fine-tuned model outscores several general-purpose vision-language models, including Janus-Pro-7B, on the same benchmark. If the claim holds, keypoint-conditioned symbolic context is an effective and cheap recipe for creating human-centric instruction data without changing model architecture.

What carries the argument

The load-bearing mechanism is the keypoint-integrated instruction data generation pipeline. For each COCO image, the paper builds a symbolic context consisting of five captions, object bounding boxes, and keypoints for each person, where each keypoint is a triple $(x, y, \text{visibility})$ covering 17 body parts from nose to ankles; this context is fed to GPT-4o, which generates the three response types without seeing the image. The resulting 200,328 samples are then used to fine-tune the entire LLaVA-1.5-7B model, and performance is measured on E-HPAUB, which extends a prior benchmark by drawing 90 COCO validation images and generating 270 conversation, description, and reasoning questions through the same keypoint prompt pipeline. The keypoints are the novel ingredient: they give the teacher explicit spatial positions of joints, which the teacher converts into natural-language statements about posture, balance, and interaction.

What would settle it

Build an E-HPAUB-style test set in which the questions are written by human annotators or by a different pipeline with a different style, scored by human judges or by a judge model other than the teacher; if LLaVA-Pose's margin over LLaVA-1.5-7B shrinks or disappears, the reported improvement is an artifact of matching the teacher's style rather than of keypoint information. A sharper test is to corrupt or randomly permute the keypoint coordinates in the training context while leaving captions and bounding boxes intact; if the model still scores 69.4 on E-HPAUB, the keypoints are not carrying the improvement.

Watch

Extended reading notes

Core claim

The central claim is that human keypoints, when integrated into the symbolic context used to generate instruction-following data, are what carry the improvement in pose and action understanding, not the captions or bounding boxes alone. The authors construct 200,328 samples from COCO by prompting GPT-4o with five captions, bounding boxes, and 17-point keypoint annotations, covering multi-turn conversation, detailed pose descriptions, and complex reasoning questions such as why an action is being performed or what technique is appropriate. After fine-tuning the full LLaVA-1.5-7B model (CLIP ViT-L vision encoder, two-layer MLP connector, and Vicuna-v1.5) for one epoch on this data, the resulting LLaVA-Pose model reaches 69.4 on E-HPAUB while the original model scores 52.1; ablated models trained on any single data type also beat the baseline, and the full mixture is best. In a comparison against DeepSeek-VL2, V*, Qwen2-VL-7B, InternVL3-8B, and Janus-Pro-7B, LLaVA-Pose achieves the highest average score of 69.6. The paper interprets this as evidence that fine-grained keypoint-level spatial information materially improves a multimodal model's interpretation of human-centric scenes.

Load-bearing premise

The results stand on the assumption that scores assigned by GPT-4o on benchmark questions that were generated by the same keypoint prompt pipeline used for the training data reflect genuine human pose and action understanding, rather than the model learning to imitate GPT-4o's descriptive style.

Editorial extensions

If this is right

  • A general-purpose vision-language model can be specialized to human-centric tasks by fine-tuning on keypoint-integrated instruction data, with no change to its architecture, reaching 69.4 versus 52.1 for the original LLaVA-1.5-7B on E-HPAUB.
  • Keypoint information is the active ingredient: the LLaVA-1.5-7B baseline, trained on captions and bounding boxes only, is outperformed by every model fine-tuned with keypoint-integrated data.
  • The three instruction data types are complementary, since the full mixture (69.4 overall) beats conversation-only (51.5), detailed-description-only (62.4), and complex-reasoning-only (65.7) training.
  • On the same benchmark, LLaVA-Pose averages 69.6 across categories, higher than DeepSeek-VL2 (48.1), V* (56.7), Qwen2-VL-7B (65.1), InternVL3-8B (66.3), and Janus-Pro-7B (68.3).
  • The improvement is limited to static images, because the model has no temporal modeling; the paper explicitly leaves video-based action understanding to future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same recipe should transfer to other structured annotations, such as facial landmarks, hand skeletons, or object keypoints, as a cheap way to produce specialized instruction data for any domain where precise spatial structure matters, since the teacher consumes only symbolic text.
  • The benchmark design leaves an open confound: because E-HPAUB questions come from the same GPT-4o keypoint pipeline and GPT-4o also scores them, part of the reported gain may be the model learning the teacher's style rather than better visual perception; a human-written, human-scored test set would separate these.
  • A testable implication of the ablation table is that keypoint-integrated data teaches mostly a vocabulary for verbalizing joint positions and body alignment, which would explain the large jump on detailed description (47.2 to 78.9 in the full-data row) and predicts smaller gains on tasks requiring causal or temporal reasoning.
  • Since the method is data-only and architecture-agnostic, combining it with temporal encoders or video instruction data is a natural next step that should extend the result to dynamic actions, which the paper names as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a method for generating vision-language instruction-following data that integrates human keypoints with captions and bounding boxes, and uses this data to fine-tune LLaVA-1.5-7B into a model called LLaVA-Pose. The authors construct a dataset of 200,328 samples across conversation, detailed description, and complex reasoning categories, and introduce an Extended Human Pose and Action Understanding Benchmark (E-HPAUB) with 270 questions. They report that LLaVA-Pose improves over the original LLaVA-1.5-7B by 33.2% on E-HPAUB, and also outperforms several recent open VLMs. The paper includes qualitative comparisons, an ablation study over data types, and a discussion of limitations related to temporal modeling.

Significance. If the central claim is correct, the keypoint-integrated data generation recipe and the released dataset could be a useful contribution to human-centric instruction tuning, and the paper provides detailed prompts and an ablation design that other researchers could build on. The authors also release code, which supports reproducibility. However, the current evaluation does not establish the claim that keypoint information is the active ingredient: the benchmark questions and the training data come from the same GPT-4o keypoint-prompt pipeline, the same GPT-4o model is used as the judge, and there is no matched no-keypoint control. The reported numeric gains may therefore partly reflect stylistic imitation of GPT-4o's pose descriptions rather than genuine improvement in pose and action understanding. The paper's scientific value is contingent on addressing these evaluation confounds, so the significance is moderate until that is done.

major comments (3)
  1. [Section 5.2, Table 8] The claim in Section 5.2 that keypoint information makes a 'significant contribution' is not supported by the experimental design: all four fine-tuned rows in Table 8 use keypoint-integrated data, and the only non-keypoint model, LLaVA-1.5-7B, was trained on a different data mix, so the observed gap conflates keypoint conditioning with the effect of fine-tuning on 200k new human-centric instruction samples. A matched control trained on the same 200,328 samples with keypoints removed (or replaced by captions and bounding boxes only) is required to isolate the keypoint contribution.
  2. [Section 5.2 and Section 3] The E-HPAUB evaluation is circular in a way that directly affects the headline 33.2% improvement: Section 5.2 states that the 270 benchmark questions are 'crafted using the data generation approach outlined in Sect. 3', which is the same GPT-4o keypoint-prompt pipeline used to create the 200,328 training samples, and GPT-4o is also the judge; consequently, a model trained to imitate GPT-4o's pose-description style can score high without improved visual perception. Please add a human evaluation with inter-annotator agreement statistics, or evaluate on an independently constructed benchmark, and report the judge's stability (for example, multiple runs or comparison with a second judge).
  3. [Tables 8 and 9] Tables 8 and 9 report incompatible sub-scores for the same model and benchmark: Table 8's Full data row gives conversation 64.3, detailed description 78.9, complex reasoning 65.0, and overall 69.4, while Table 9's LLaVA-Pose row gives 77.4, 58.6, 72.9, and overall 69.6; this internal inconsistency suggests the GPT-4o scoring protocol is not stable enough to support precise numeric claims such as '33.2% increase' and '1.3-point margin'. Please explain the discrepancy and report the evaluation protocol (prompt, temperature, number of runs, aggregation method) together with variance or confidence intervals, and do not present single-run point estimates from a stochastic judge as exact.
minor comments (4)
  1. [Throughout] There are several typos and formatting issues, including 'T able' in table captions, 'PmLr' in reference 25, 'V ∗' rendered inconsistently, and 'about regarding' in Section 4; these should be corrected in a final pass.
  2. [Section 5.2] The paper should describe how the 90 E-HPAUB images are selected and how the three questions per image are generated, because this information is needed for reproducibility and for assessing whether the benchmark is representative.
  3. [Section 5.2] The scoring rubric mentioned in Section 5.2 (helpfulness, relevance, accuracy, detail) is described only in prose; please provide the exact prompt and scoring criteria used to query GPT-4o, so that readers can replicate the evaluation.
  4. [Section 5.2 and Reference [36]] Since E-HPAUB is an extension of the authors' own HPAUB from reference [36], the paper should clarify exactly what E-HPAUB adds beyond HPAUB and why the extension is necessary for this study; this will also help readers judge the degree of novelty relative to the prior work.

Circularity Check

2 steps flagged · score 6.0 of 10

E-HPAUB questions and training targets come from the same GPT-4o keypoint pipeline, and GPT-4o also judges the answers; the reported 33.2% gain therefore measures stylistic imitation of the teacher more than independently established pose understanding.

  1. self definitional [Section 5.2 (Quantitative Evaluation), E-HPAUB construction; cross-referenced to Section 3 data generation]
    "The questions are crafted using the data generation approach outlined in Sect.3."

    The training data in Section 3 is generated by prompting GPT-4o with COCO captions, bounding boxes, and keypoints ("using GPT-4o [23] as a teacher model"), producing the same three response types: conversation, detailed description, and complex reasoning. E-HPAUB's 270 test questions are explicitly "crafted using the data generation approach outlined in Sect.3," so the test distribution is produced by the same teacher, from the same symbolic keypoint context, and with the same response taxonomy.

  2. other [Section 5.2 (Quantitative Evaluation), evaluation protocol]
    "A language-only GPT-4o then evaluates these responses on a scale of 1-10, considering helpfulness, relevance, accuracy, and detail level."

    The same model family that produced the training answers and the benchmark questions also assigns the evaluation scores. The "ground-truth descriptions" in the image-ground-truth-question triplets are of the same kind generated by the Section 3 keypoint pipeline, so there is no external ground truth or human-judged anchor for the 69.4 vs. 52.1 gap. The scoring rewards outputs that resemble GPT-4o's characteristic pose vocabulary (bent knees, lowered center of gravity, counterbalancing arms), closing a loop: GPT-4o generates the target style, the model is fine-tuned on that style, and GPT-4o then judges adherence to that same style.

full rationale

The central claim of a 33.2% improvement in human pose and action understanding is not derived from an external benchmark; it rests on E-HPAUB, whose questions are generated by the same GPT-4o keypoint-conditioned prompt pipeline used to create the 200,328 training samples, and whose scores are assigned by GPT-4o. That is a concrete self-consistency loop, not just a suspicion: the paper states both that training uses GPT-4o as teacher (Sect. 3) and that E-HPAUB questions are "crafted using the data generation approach outlined in Sect.3" (Sect. 5.2), with GPT-4o also serving as judge. I score this 6 rather than higher because the training and evaluation images are disjoint (COCO train vs. COCO validation) and the ablations show some sensitivity to data type, so the improvement is not entirely forced by identical data. I did not count as circular the self-citation of HPAUB [36]; it is the shared GPT-4o pipeline that drives the loop. Separately, the paper lacks a matched no-keypoint control: Table 8 compares against the original LLaVA-1.5-7B, which was trained on a different data mix, so the claim that keypoints specifically are the active ingredient is a confound rather than a circularity. The evaluation protocol also appears unstable (Table 8 Full data sub-scores 64.3/78.9/65.0 vs. Table 9 LLaVA-Pose sub-scores 77.4/58.6/72.9 for the same model and benchmark), which further weakens the precise numeric claim but is not itself a circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted mathematical parameters are present; the empirical claim depends on four domain assumptions about annotation quality, teacher-model fidelity, GPT-4o judging validity, and benchmark representativeness. The absence of a human-validated external benchmark is the main burden.

assumptions (4)
  • domain assumption COCO caption, bounding box, and keypoint annotations are accurate and sufficient for describing human pose and action.
    These annotations are the only visual signal given to GPT-4o during data generation in Section 3, so any annotation errors propagate into both training and benchmark data.
  • domain assumption GPT-4o can generate faithful instruction-following data from symbolic context without seeing the image.
    The 200,328 generated samples are treated as ground truth for fine-tuning in Sections 3 and 5, with no human verification of factual accuracy.
  • domain assumption GPT-4o evaluation scores reflect human pose/action understanding quality.
    All quantitative claims in Tables 8 and 9 rely on GPT-4o as judge, and no human agreement study or correlation with human judgment is reported.
  • domain assumption E-HPAUB's 90 images and 270 questions are representative of human pose/action understanding.
    The benchmark is constructed by random selection from COCO val plus GPT-4o-generated questions, with no evidence that this small set covers the difficulty range of pose/action understanding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLaVA-Pose: Enhancing Human Pose and Action Understanding via Keypoint-Integrated Instruction Tuning." pith.science (2026). https://pith.science/paper/JIEI74CR

@misc{pith2026250621317,
  author       = {Pith},
  title        = {Pith review of: LLaVA-Pose: Enhancing Human Pose and Action Understanding via Keypoint-Integrated Instruction Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JIEI74CR}},
  note         = {Machine review of arXiv:2506.21317}
}
read the original abstract

Current vision-language models (VLMs) are well-adapted for general visual understanding tasks. However, they perform inadequately when handling complex visual tasks related to human poses and actions due to the lack of specialized vision-language instruction-following data. We introduce a method for generating such data by integrating human keypoints with traditional visual features such as captions and bounding boxes, enabling more precise understanding of human-centric scenes. Our approach constructs a dataset comprising 200,328 samples tailored to fine-tune models for human-centric tasks, focusing on three areas: conversation, detailed description, and complex reasoning. We establish an Extended Human Pose and Action Understanding Benchmark (E-HPAUB) to assess model performance on human pose and action understanding. We fine-tune the LLaVA-1.5-7B model using this dataset and evaluate our resulting LLaVA-Pose model on the benchmark, achieving significant improvements. Experimental results show an overall improvement of 33.2% compared to the original LLaVA-1.5-7B model. These findings highlight the effectiveness of keypoint-integrated data in enhancing multimodal models for human-centric visual understanding. Code is available at https://github.com/Ody-trek/LLaVA-Pose.

Figures

Figures reproduced from arXiv: 2506.21317 by the authors.

Figure 1
Figure 1. One example to demonstrate the structure of instruction-following data. The top block displays the contexts information, including captions, bounding boxes (shown as solid rectangles in the visual image) and keypoints (shown as green circular mark￾ers in the visual image) used to prompt GPT-4o, and the bottom block displays the three types of responses generated. It is important to note that the visual image itself … view at source ↗
Figure 2
Figure 2. LLaVA-Pose model architecture. 5.1 Qualitative Evaluation We conduct a qualitative evaluation to compare the responses of seven models: DeepSeek-VL2 [34], V ∗ [33], Qwen2-VL-7B [29], InternVL3-8B [40], Janus-Pro￾7B [5], LLaVA-1.5-7B [19], and our LLaVA-Pose [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Superman: Unifying Skeleton and Vision for Human Motion Perception and Generation

    cs.CV 2026-02 conditional novelty 6.0 of 10

    A single MLLM trained with a vision-guided hybrid VQ-VAE tokenizer reports state-of-the-art or competitive results for 3D pose estimation, motion prediction, and motion in-betweening on Human3.6M and 3DPW.

  2. PoseLLM: Enhancing Language-Guided Human Pose Estimation with MLP Alignment

    cs.CV 2025-07 conditional novelty 3.0 of 10

    PoseLLM swaps LocLLM's linear vision-language projector for a two-layer MLP with GELU, reporting +0.4 AP on COCO (77.8) with comparable zero-shot transfer.

Reference graph

Works this paper leans on

40 extracted references · 20 canonical work pages · cited by 2 Pith papers

  1. [1]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Advances in neural information processing systems35, 23716–23736 (2022)

    Alayrac, J.B., et al.: Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems35, 23716–23736 (2022)

    work page 2022

  3. [3]

    Byeon, M., Park, B., Kim, H., Lee, S., Baek, W., Kim, S.: Coyo-700m: Image-text pair dataset (2022),https://github.com/kakaobrain/coyo-dataset, accessed 5 Apr 2025

    work page 2022

  4. [4]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Changpinyo, S., Sharma, P., Ding, N., Soricut, R.: Conceptual 12m: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3558–3568 (2021)

    2021

  5. [5]

    arXiv preprint arXiv:2501.17811 (2025)

    Chen, X., Wu, Z., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C.: Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811 (2025)

    arXiv 2025

  6. [6]

    Chiang, W.L., et al.: Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality (2023),https://lmsys.org/blog/2023-03-30-vicuna/, accessed 5 Apr 2025

    work page 2023

  7. [7]

    In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision

    Cui, C., et al.: A survey on multimodal large language models for autonomous driving. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 958–979 (2024)

    work page 2024

  8. [8]

    In: European Conference on Computer Vision

    Delmas, G., Weinzaepfel, P., Lucas, T., Moreno-Noguer, F., Rogez, G.: Posescript: 3d human poses from natural language. In: European Conference on Computer Vision. pp. 346–362. Springer (2022)

    work page 2022

Show all 40 references
  1. [9]

    IEEE transactions on pattern anal- ysis and machine intelligence (2024)

    Delmas, G., Weinzaepfel, P., Lucas, T., Moreno-Noguer, F., Rogez, G.: Posescript: Linking 3d human poses and natural language. IEEE transactions on pattern anal- ysis and machine intelligence (2024)

    2024

  2. [10]

    In: Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision

    Delmas, G., Weinzaepfel, P., Moreno-Noguer, F., Rogez, G.: Posefix: correcting 3d human poses with natural language. In: Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision. pp. 15018–15028 (2023)

    2023

  3. [11]

    In: Proceedings of the 40th International Conference on Machine Learning

    Driess, D., Xia, F., Sajjadi, M.S.M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., Zeng, A., Mordatch, I., Florence, P.: ...

    2023

  4. [12]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Feng, D., Guo, P., Peng, E., Zhu, M., Yu, W., Wang, P.: Posellava: Pose centric multimodal llm for fine-grained 3d pose manipulation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 2951–2959 (2025)

    2025

  5. [13]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Feng, Y., Lin, J., Dwivedi, S.K., Sun, Y., Patel, P., Black, M.J.: Chatpose: Chatting about 3d human pose. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2093–2103 (2024)

    2024

  6. [14]

    IEEE Transactions on Instru- mentation and Measurement72, 1–12 (2023) 16

    Kyrollos, D.G., Fuller, A., Greenwood, K., Harrold, J., Green, J.R.: Under the cover infant pose estimation using multimodal data. IEEE Transactions on Instru- mentation and Measurement72, 1–12 (2023) 16

    2023

  7. [15]

    Mul- timedia Tools and Applications82(14), 20771–20818 (2023)

    Le, V.H.: Deep learning-based for human segmentation and tracking, 3d human pose estimation and action recognition on monocular video of mads dataset. Mul- timedia Tools and Applications82(14), 20771–20818 (2023)

    2023

  8. [16]

    In: International conference on machine learning

    Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: International conference on machine learning. pp. 19730–19742. PMLR (2023)

    2023

  9. [17]

    ACM Computing Sur- veys56(10), 1–42 (2024)

    Liang, P.P., Zadeh, A., Morency, L.P.: Foundations & trends in multimodal ma- chine learning: Principles, challenges, and open questions. ACM Computing Sur- veys56(10), 1–42 (2024)

    2024

  10. [18]

    In: Computer vision– ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13

    Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer vision– ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13. pp. 740–755...

    2014

  11. [19]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 26296–26306 (2024)

    2024

  12. [20]

    Advances in neural information processing systems36, 34892–34916 (2023)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems36, 34892–34916 (2023)

    2023

  13. [21]

    In: Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp

    Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: Smpl: A skinned multi-person linear model. In: Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 851–866 (2023)

    2023

  14. [22]

    Sensors23(4), 2182 (2023)

    Morshed, M.G., Sultana, T., Alam, A., Lee, Y.K.: Human action recognition: A taxonomy-based survey, updates, and opportunities. Sensors23(4), 2182 (2023)

    2023

  15. [23]

    OpenAI: Hello gpt-4o (2024),https://openai.com/index/hello-gpt-4o/, ac- cessed 5 Apr 2025

    2024

  16. [24]

    arXiv preprint arXiv:2306.14824 (2023)

    Peng, Z., et al.: Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824 (2023)

    2023 arXiv

  17. [25]

    In: International conference on machine learning

    Radford, A., et al.: Learning transferable visual models from natural language su- pervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

    2021

  18. [26]

    In: Proceed- ings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining

    Rasley, J., Rajbhandari, S., Ruwase, O., He, Y.: Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In: Proceed- ings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. pp. 3505–3506 (2020)

    2020

  19. [27]

    Advances in neural information processing systems 35, 25278–25294 (2022)

    Schuhmann, C., et al.: Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems 35, 25278–25294 (2022)

    2022

  20. [28]

    Sensors23(14), 6384 (2023)

    Surek, G.A.S., et al.: Video-based human activity recognition using deep learning approaches. Sensors23(14), 6384 (2023)

    2023

  21. [29]

    arXiv preprint arXiv:2409.12191 (2024)

    Wang, P., et al.: Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024)

    2024 arXiv

  22. [30]

    Advanced Intelligent Systems6(3), 2300359 (2024)

    Wang, T., Zheng, P., Li, S., Wang, L.: Multimodal human–robot interaction for human-centric smart manufacturing: a survey. Advanced Intelligent Systems6(3), 2300359 (2024)

    2024

  23. [31]

    Advances in Neural Information Processing Systems36, 61501–61513 (2023)

    Wang, W., et al.: Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems36, 61501–61513 (2023)

    2023

  24. [32]

    In: 2023 IEEE International Conference on Big Data (BigData)

    Wu, J., Gan, W., Chen, Z., Wan, S., Yu, P.S.: Multimodal large language models: A survey. In: 2023 IEEE International Conference on Big Data (BigData). pp. 2247–2256. IEEE (2023) 17

    2023

  25. [33]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wu, P., Xie, S.: V?: Guided visual search as a core mechanism in multimodal llms. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13084–13094 (2024)

    2024

  26. [34]

    Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., Wang, B., Xie, Z., Wu, Y., Hu, K., Wang, J., Sun, Y., Li, Y., Piao, Y., Guan, K., Liu, A., Xie, X., You, Y., Dong, K., Yu, X., Zhang, H., Zhao, L., Wang, Y., Ruan, C.: Deepseek-vl2: Mixture-of-exper...

    2024 arXiv

  27. [35]

    In: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

    Yildirim, N., et al.: Multimodal healthcare ai: identifying and designing clinically relevant vision-language applications for radiology. In: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. pp. 1–22 (2024)

    2024

  28. [36]

    Zhang, D., An, W., Shouno, H.: Keypoint-integrated instruction-following data generation for enhanced human pose and action understanding in multimodal mod- els (2025),https://arxiv.org/abs/2409.09306

    2025 arXiv

  29. [37]

    ACM Computing Surveys56(1), 1–37 (2023)

    Zheng, C., et al.: Deep learning-based human pose estimation: A survey. ACM Computing Surveys56(1), 1–37 (2023)

    2023

  30. [38]

    Electronics12(13), 2805 (2023)

    Zhou, H., Wang, D., Yu, Y., Zhang, Z.: Research progress of human–computer in- teraction technology based on gesture recognition. Electronics12(13), 2805 (2023)

    2023

  31. [39]

    arXiv preprint arXiv:2304.10592 (2023)

    Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 (2023)

    2023 arXiv

  32. [40]

    Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y., Su, W., Shao, J., Gao, Z., Cui, E., Wang, X., Cao, Y., Liu, Y., Wei, X., Zhang, H., Wang, H., Xu, W., Li, H., Wang, J., Deng, N., Li, S., He, Y., Jiang, T., Luo, J., Wang, Y., He, C., Shi, B., Zhang, X.,...

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.