Pith. sign in

REVIEW 5 major objections 5 minor 72 references

An LLM-Empowered Low-Resolution Vision System for On-Device Human Behavior Understanding

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a three-stage pipeline of contrastive pseudo-labeling, consistency-checked LLM captioning, and LoRA tuning lets LVLMs understand low-resolution human-behavior video, reaching up to 87.02% Bert-Score F1.

desk verdict Useful systems paper with a real deployment; headline public-dataset numbers are undercut by a synthetic-reference evaluation that risks measuring label leakage rather than caption quality. read the letter →

arxiv 2505.01743 v1 pith:35K2YQUJ submitted 2025-05-03 cs.CV cs.AIcs.LG

Siyang Jiang , Bufang Yang , Lilin Xu , Mu Yuan , Yeerzhati Abudunuer , Kaiwei Liu , Liekang Zeng , Hongkai Chen
show 3 more authors
Zhenyu Yan Xiaofan Jiang Guoliang Xing
This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords low-resolutionvisionhumanbehaviorunderstandingvideocaptioningpseudo-labelingcontrastivelearninglargelanguagemodelsLoRAfine-tuningedgecomputing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Llambda is a three-stage system that claims to let large vision-language models (LVLMs) caption human behavior in low-resolution video—depth, thermal, or infrared—using almost no manual annotation. The paper's central claim is that class-aware guidance is the missing ingredient: a contrastively trained labeler turns a little labeled data plus a lot of unlabeled data into reliable pseudo-labels, spatial and temporal consistency checks clean those labels, and an LLM then writes captions that a LoRA-tuned LVLM learns from. On three public datasets and a 97-node real-world depth testbed, the paper reports up to 87.02% Bert-Score F1 and an average 40.03% relative improvement over five state-of-the-art LVLM baselines, with only 1% of the testbed data labeled. If this holds, it would remove the main cost barrier—manual annotation at roughly $800 per six hours of video—to privacy-preserving in-home monitoring for dementia and rehabilitation care.

What carries the argument

The load-bearing mechanism is the class-aware pseudo-label used as an intermediate representation between sensor frames and language. A window-based sensitivity filter drops empty frames using cheap pixel-difference statistics without running a continuous detector, a YOLO-V8 detector crops the human subject (fine-tuned on just 1% of data when the resolution is very low), and a ResNet18 labeler trained with a semantic-aware NT-Xent contrastive loss plus cross-entropy outputs per-frame top-k action distributions. The Physical-Knowledge-Guided Captioner then applies intra-distribution spatial and inter-distribution temporal consistency checks—removing, for example, a 'running' label inside a 'sleeping' sequence—and hands the cleaned top-k action scores to Llama-3.1-70B, which writes the video-level caption. Finally, Qwen2.5-VL-7B is fine-tuned on the resulting low-resolution-video/caption pairs with rank-8 LoRA, using Q-LoRA to fit the 17.8 GB training memory footprint onto a Jetson Xavier NX; trainable parameter overhead stays below 3%.

What would settle it

Re-score Llambda against the strongest baseline on UTD-D using human-written reference captions created by annotators who see only the low-resolution video; if the Bert-Score advantage largely disappears or reverses, the improvement is an artifact of matching the Tarsier-generated reference distribution. A second check: feed the LLM top-k action scores replaced by uniform or shuffled distributions; if the Bert-Score stays flat, the pseudo-labeler is not the source of the gain.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that the low-resolution gap in LVLMs is not a resolution problem but an information-delivery problem: the same 7B LVLM that fails on raw depth frames produces high-quality captions once fine-tuned on captions generated under class-aware guidance. Concretely, the paper shows that captions outperform labels when fine-tuning (69.71 versus 53.40 Bert-Score for Qwen-VL), that contrastive pseudo-labeling with consistency checks can replace human annotation, and that the resulting pipeline beats Qwen2.5-VL, ImageBind-LLM, OneLLM, Language-Bind, and Tarsier across all three low-resolution modalities. The claim is deliberately framed as a system result: each stage contributes, and the whole works under 1% supervision and on edge hardware.

Load-bearing premise

The evaluation assumes that the reference captions used for scoring—Tarsier-34B captions of RGB video supplemented with manual action descriptions on the public datasets, and three volunteers' captions on the testbed—are a correct and unbiased standard for what a low-resolution caption should say; if that reference is biased or incomplete, the reported Bert-Score gaps do not measure genuine caption quality.

Editorial extensions

If this is right

  • Manual annotation stops being the bottleneck for human behavior understanding: with only 1% labeled data on the real-world testbed, Llambda still outperforms all five baselines, so the pipeline can scale to long-running in-home deployments without repeated $800-per-six-hours labeling.
  • Privacy-preserving sensing becomes caption-capable: depth, thermal, and infrared streams, which blur identifiable details, can now support detailed behavior understanding rather than only coarse activity labels.
  • On-device personalization is feasible: LoRA plus Q-LoRA keeps trainable parameters under 3% and fits fine-tuning on edge hardware, so models can adapt locally to a user's home and routines without shipping raw video to a cloud.
  • Caption-based supervision beats label-based supervision for LVLMs in this domain, implying that HBU systems should be built as caption-generation tasks rather than classification tasks.
  • The same class-aware-guidance recipe is claimed to extend beyond vision to other sparse modalities such as IMU and point cloud data, per the paper's discussion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit: because the gain is attributed to class-aware guidance, prompting the same LVLM with the action taxonomy but no fine-tuning should already narrow part of the gap; measuring that headroom would isolate the contribution of the LoRA stage.
  • A natural next step the authors do not develop: the large captioning LLM (Llama-3.1-70B) is only needed offline to produce training captions; distilling those captions into the small LoRA-tuned LVLM would let the deployed edge node caption without the 70B model at inference.
  • The consistency checks encode physical priors about how actions persist over time; extending them to multi-person scenes, the paper's stated limitation, would likely require representing pseudo-labels as a graph or a set of tracks rather than per-frame top-k lists.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Llambda, a three-stage system for generating captions of human behavior from low-resolution videos (depth, thermal, infrared). The first stage trains a contrastive labeler that produces class-aware pseudo-labels for unlabeled data, after filtering frames with a window-based sensitivity mechanism and cropping human regions. The second stage uses spatial and temporal consistency checks to convert a time series of pseudo-labels into prompts for an LLM (Llama-3.1-70B), producing video-level captions. The third stage fine-tunes Qwen2.5-VL-7B with LoRA on the resulting video-caption pairs. The authors evaluate on a 97-node real-world depth testbed and on three public datasets (UTD-D, IM-T, SC-IR), reporting Bert-Score F1 up to 87.02% and relative improvements up to 40.03% over several LVLM baselines.

Significance. If the reported results are sound, Llambda would be a useful systems contribution: it would show that class-aware pseudo-labels and consistency-checked LLM captions can substantially close the gap between general LVLMs and low-resolution human behavior understanding, while keeping on-device training feasible via LoRA. The paper has concrete strengths: a real-world testbed with 97 distributed nodes and human volunteer annotations for the RW-D data, evaluation across three low-resolution modalities, and explicit attention to deployability with LoRA/Q-LoRA. However, the quantitative claims are currently not fully supported by the evaluation protocol, as detailed in the major comments; the headline margins on public datasets rest on synthetic references that may share vocabulary with Llambda's pseudo-label classes.

major comments (5)
  1. [§6.1.2, §6.2.1] The public-dataset ground truth for UTD-D, IM-T, and SC-IR is generated by Tarsier-34B from RGB video and then supplemented with action descriptions. Because Llambda's captioning stage is explicitly driven by class-aware pseudo-labels from the contrastive labeler, its outputs will naturally contain the action vocabulary used in the reference, whereas the baselines generate captions directly from low-resolution video without this class information. Under Bert-Score, which measures token/semantic overlap, this setup can reward vocabulary overlap rather than true caption quality for low-resolution inputs. The paper needs to validate the synthetic references, for example by obtaining human references on a subsample and reporting scores against those, and should also discuss whether the supplement step introduces a systematic advantage for Llambda.
  2. [§5.3, §6.2.1, §6.2.4] All performance comparisons are reported as point estimates without error bars, confidence intervals, or significance tests. Figures 10, 12, 15, and 16 show no measure of run-to-run variability, and the text does not state the number of independent runs or seeds. Without this information, it is impossible to assess whether the reported margins, especially the 40.03% relative improvement, are statistically meaningful. This is a load-bearing issue because the central claim is a quantitative superiority over baselines.
  3. [§5.2.1, §6.1.3] The evaluation protocol for the baselines is underspecified. The paper does not state whether Qwen, ImageBind-LLM, OneLLM, Language-Bind, and Tarsier are evaluated zero-shot or after any fine-tuning, nor does it specify the input modality, prompting, or hyperparameters used for them. Since Llambda is fine-tuned on low-resolution data with LoRA, a comparison against zero-shot baselines would conflate the benefit of fine-tuning with the benefit of the proposed pipeline. The authors should specify the exact protocol for every baseline and ideally include a fine-tuned baseline without the pseudo-label/captioning stages.
  4. [§6.1.3, §6.2.4] Top-K selection is performed after observing results on the UTD-D test set; the text states that top-5 and top-3 outperform top-1 and then presents top-5/top-3 as the final configuration. This is selection on the test data and can overstate performance. A held-out validation split or a nested procedure is needed. Relatedly, the paper drops 'extremely minority classes' from IM-T without specifying which classes or how many, which changes the evaluation set and makes cross-dataset comparisons difficult to interpret.
  5. [§4.2, §4.1] The claimed contributions of the spatial and temporal consistency checks and of the contrastive labeler are not separately quantified. An ablation that removes intra-distribution checking, inter-distribution checking, the contrastive loss, or the pseudo-label guidance would help establish that these components, rather than the overall fine-tuning procedure, are responsible for the reported gains. Without such ablations, the causal claim behind the system design is not fully supported.
minor comments (5)
  1. [§6.2.2] The text says the filtering experiment is evaluated on the IM-T dataset, while the caption of Figure 13 says 'on UTD-D'; these should be reconciled.
  2. [Table 1, §5.1] Table 1 lists 7856 'labeled videos' for RW-D, but the text states that only 1% of collected data is labeled; the table heading and the relationship between total videos and labeled videos should be clarified.
  3. [Throughout] There are several typographical errors, including 'T arsier' in Figure 12, 'he UTD-D' in the §6.2.2 footnote, 'detail implemtation' in §5.2.2, and 'Guoling Xing' for the corresponding author. A careful proofreading pass is needed.
  4. [§4.1.1, §4.1.2, §4.1.3] Several hyperparameters are introduced but their values are not reported: sigma in Eq. (1), epsilon in §4.1.2, lambda and tau in Eq. (2)-(3). Reporting the chosen values is important for reproducibility.
  5. [§5.2.2] The RW-D ground truth is annotated by three volunteers, but no inter-annotator agreement measure is reported; this would strengthen confidence in the reference captions.

Circularity Check

1 steps flagged · score 2.0 of 10

No equation-level circularity; central claim rests on empirical evaluation. One minor benchmark-construction concern: public-dataset ground truth is built from Tarsier captions plus action supplements, which aligns with Llambda's pseudo-label inputs, but the human-annotated RW-D testbed provides independent grounding.

  1. other [Section 6.1.2 (Data caption generation); scoring in §6.2.1]
    "For public datasets, we employed Tarsier-34B, to generate captions for the RGB data, as these models are known to perform well on this modality. However, because these models do not accurately capture human behavior, we supplemented their outputs with precise descriptions of actions, forming our final evaluation set."

    The public-dataset reference is constructed by adding action-description supplements to Tarsier captions, while Llambda's captioner is prompted with class-aware pseudo-labels, i.e., top-K action predictions (§4.2.1, §4.2.3). Bert-Score against this reference therefore rewards the same action vocabulary that Llambda receives as input, so the public-dataset margins partly encode the supplement rather than independent caption quality. This is a benchmark-construction alignment rather than an equation-level derivation, and it is mitigated by the human-annotated RW-D testbed (§5.2.2), so the circularity burden is low.

full rationale

The claimed contribution is an engineered pipeline evaluated empirically, not a derived equation. The labeler loss (Eq. 2-3) combines standard NT-Xent contrastive loss with cross-entropy; the captioner is an LLM prompted with top-K pseudo-labels; the LoRA fine-tuning is standard low-rank adaptation. None of these steps fits a parameter to the reported Bert-Score targets, and no same-author citation is load-bearing: ArtFL, DrHouse, SocialMind, and PGADA appear as related-work or optional aggregation choices rather than as justification for the central result. The one notable self-reference is the construction of the public-dataset reference in §6.1.2, where Tarsier-34B captions are supplemented with action descriptions and Llambda's prompted pseudo-labels carry the same action vocabulary; this makes the UTD-D/IM-T/SC-IR Bert-Score comparisons partly reward input-label overlap. This is a benchmark-construction issue rather than derivation-level circularity, and the RW-D testbed's volunteer annotations (§5.2.2) provide independent support. Overall, no significant circularity is present; the minor evaluation-construction concern keeps the score at 2 rather than 0.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The central claim does not rest on a closed-form derivation; it rests on engineering choices and evaluation assumptions. The free parameters are mostly hyperparameters, with top-K being the one tuned directly on an evaluation dataset. The axioms are domain assumptions about low-resolution sensing and about the validity of model-generated ground truth.

free parameters (8)
  • sigma = not reported
    Threshold in Eq. 1 deciding whether a pixel difference is significant; chosen by hand with no sensitivity analysis reported.
  • N = 2
    Minimum number of significant frame differences required to retain a frame, set in §4.1.1.
  • epsilon = not reported
    Maximum bounding-box displacement between adjacent frames for temporal coherence in §4.1.2.
  • lambda = not reported
    Balance between contrastive loss and cross-entropy loss in Eq. 3.
  • tau = not reported
    Temperature in the NT-Xent contrastive loss, Eq. 2.
  • top-K = 5
    Number of pseudo-label categories per frame sent to the LLM; selected based on UTD-D experiments in §6.2.4 rather than a separate validation set.
  • LoRA rank = 8
    Rank of LoRA decomposition in §6.1.3; no ablation on rank is provided.
  • labeled data fraction = 1%
    Amount of collected data used to fine-tune YOLO and train the labeler in the real-world testbed; this ratio is a design choice.
assumptions (4)
  • domain assumption Human presence can be distinguished from sensor noise by the temporal profile of pixel differences, with gradual changes indicating entry and abrupt spikes indicating noise.
    Used in §4.1.1 to justify the window-based sensitivity filter; if false, the filter drops behavior-relevant frames or retains noise.
  • domain assumption A pretrained YOLO detector, optionally fine-tuned on 1% labeled data, reliably localizes people in low-resolution depth, thermal, and infrared frames.
    Invoked in §4.1.2; the cropping mechanism depends on accurate boxes, and only qualitative and one-point quantitative evidence is given.
  • domain assumption Tarsier-34B captions of RGB video, supplemented by manual action descriptions, form a valid ground truth for low-resolution caption evaluation.
    Stated in §6.1.2; all public-dataset Bert-Score numbers depend on this reference.
  • domain assumption Bert-Score F1 is an adequate proxy for caption quality in HBU.
    Used as the sole evaluation metric in §5.2.2; no human evaluation of final captions is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An LLM-Empowered Low-Resolution Vision System for On-Device Human Behavior Understanding." pith.science (2026). https://pith.science/paper/35K2YQUJ

@misc{pith2026250501743,
  author       = {Pith},
  title        = {Pith review of: An LLM-Empowered Low-Resolution Vision System for On-Device Human Behavior Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/35K2YQUJ}},
  note         = {Machine review of arXiv:2505.01743}
}
abstract

The rapid advancements in Large Vision Language Models (LVLMs) offer the potential to surpass conventional labeling by generating richer, more detailed descriptions of on-device human behavior understanding (HBU) in low-resolution vision systems, such as depth, thermal, and infrared. However, existing large vision language model (LVLM) approaches are unable to understand low-resolution data well as they are primarily designed for high-resolution data, such as RGB images. A quick fixing approach is to caption a large amount of low-resolution data, but it requires a significant amount of labor-intensive annotation efforts. In this paper, we propose a novel, labor-saving system, Llambda, designed to support low-resolution HBU. The core idea is to leverage limited labeled data and a large amount of unlabeled data to guide LLMs in generating informative captions, which can be combined with raw data to effectively fine-tune LVLM models for understanding low-resolution videos in HBU. First, we propose a Contrastive-Oriented Data Labeler, which can capture behavior-relevant information from long, low-resolution videos and generate high-quality pseudo labels for unlabeled data via contrastive learning. Second, we propose a Physical-Knowledge Guided Captioner, which utilizes spatial and temporal consistency checks to mitigate errors in pseudo labels. Therefore, it can improve LLMs' understanding of sequential data and then generate high-quality video captions. Finally, to ensure on-device deployability, we employ LoRA-based efficient fine-tuning to adapt LVLMs for low-resolution data. We evaluate Llambda using a region-scale real-world testbed and three distinct low-resolution datasets, and the experiments show that Llambda outperforms several state-of-the-art LVLM systems up to $40.03\%$ on average Bert-Score.

Figures

Figures reproduced from arXiv: 2505.01743 by the authors.

Figure 1
Figure 1. Limitation of labels. The left depth frame il [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Overview of Llambda. Llambda has three stages including Contrastive-Oriented Data Labeler, Physical [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 5
Figure 5. Motivation of dynamic action capturing. We [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figures from the paper (8 more)
Figure 6
Figure 6. Figure 6: Illustration of intra- and inter-distribution [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Data example collected from our testbed and [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: The distribution of labeled data of our testbed. [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: (Left): Performance (Bert-Score) comparison on RW-D testbed. Llambda consistently outperforms the state-of-the-art baselines. (Right): System perfor￾mance results of Llambda, including average computa￾tion, communication, and waiting time (in minutes). flexible alignm…
Figure 11
Figure 11. Figure 11: Illustration example of our self-collected [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]
Figure 12
Figure 12. Figure 12: Performance (Bert-Score) comparison on UTD-D, IM-T, SC-IR datasets. Llambda consistently outperforms [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 13
Figure 13. Figure 13: Results of window-based sensitivity on￾camera filtering across three baselines on UTD-D. Thermal w/o FT Depth w/o FT Depth w FT Accuracy Percentage [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]
Figure 16
Figure 16. Figure 16: (Left): Bert-Score across varying amounts of labeled data. (Right): Accuracy of Labeler across varying amounts of labeled data. and SC-IR datasets under different proportions of labeled data (30%, 60%, and 100%). As shown in the left part of Fig.16, we observe that fo…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 46 canonical work pages

  1. [1]

    Asuna Arai, Amartuvshin Khaltar, Takashi Ozaki, and Yuriko Kat- sumata. 2021. Influence of social interaction on behavioral and psy- chological symptoms of dementia over 1 year among long-term care facility residents. Geriatric nursing 42, 2 (2021), 509–516

    work page 2021

  2. [2]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)

    arXiv 2023

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL Technical Rep...

    arXiv 2025

  4. [4]

    Chen Chen, Roozbeh Jafari, and Nasser Kehtarnavaz. 2015. UTD- MHAD: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor. In2015 IEEE International conference on image processing (ICIP) . IEEE, 168–172

    work page 2015

  5. [5]

    Guikun Chen and Wenguan Wang. 2024. A survey on 3d gaussian splatting. arXiv preprint arXiv:2401.03890 (2024)

    arXiv 2024

  6. [6]

    Richard Chen, Filip Jankovic, Nikki Marinsek, Luca Foschini, Lampros Kourtis, Alessio Signorini, Melissa Pugh, Jie Shen, Roy Yaari, Vera Maljkovic, et al. 2019. Developing measures of cognitive impairment in the real world from consumer-grade multimodal sensor streams. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery &...

    work page 2019

  7. [7]

    Xi Chen, Julien Cumin, Fano Ramparany, and Dominique Vaufreydaz

    work page

  8. [8]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al . 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 24185–24198

    2024

Show all 72 references
  1. [9]

    Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. 2023. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 9 (2023), 10850–10869

    2023

  2. [10]

    Weichen Dai, Yu Zhang, Shenzhou Chen, Donglei Sun, and Da Kong

  3. [11]

    Iceberge Data. 2025. Iceberge Data. https://mark.hk.cn/pricing/

    2025

  4. [12]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer

  5. [13]

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. Im- agebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 15180–15190

    2023

  6. [14]

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. 2023. OneLLM: One Framework to Align All Modalities with Language. arXiv preprint arXiv:2312.03700 (2023)

    2023 arXiv

  7. [15]

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. 2024. Onellm: One framework to align all modalities with language. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 26584–26595

    2024

  8. [16]

    Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. 2023. Imagebind- llm: Multi-modality instruction tuning. arXiv preprint arXiv:2309.03905 (2023)

    2023 arXiv

  9. [17]

    Kaiyuan Hou, Yanchen Liu, Peter Wei, Chenye Yang, Hengjiu Kang, Stephen Xia, Teresa Spada, Andrew Rundle, and Xiaofan Jiang. 2022. A low-cost in-situ system for continuous multi-person fever screening. In 2022 21st ACM/IEEE International Conference on Information Processing in...

    2022

  10. [18]

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. [n.d.]. LoRA: Low-Rank Adapta- tion of Large Language Models. InInternational Conference on Learning Representations

  11. [19]

    Sijie Ji, Xinzhe Zheng, and Chenshu Wu. 2024. Hargpt: Are llms zero- shot human activity recognizers?. In 2024 IEEE International Workshop on Foundation Models for Cyber-Physical Systems & Internet of Things (FMSys). IEEE, 38–43

    2024

  12. [20]

    Siyang Jiang, Wei Ding, Hsi-Wen Chen, and Ming-Syan Chen. 2022. PGADA: Perturbation-guided adversarial alignment for few-shot learn- ing under the support-query shift. In Pacific-Asia Conference on Knowl- edge Discovery and Data Mining . Springer, 3–15

    2022

  13. [21]

    Siyang Jiang, Xian Shuai, and Guoliang Xing. 2024. ArtFL: Exploiting data resolution in federated learning for dynamic runtime inference via multi-scale training. In 2024 23rd ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) . IEEE, 27–38

    2024

  14. [22]

    2023.Ultralytics YOLOv8

    Glenn Jocher, Ayush Chaurasia, and Jing Qiu. 2023.Ultralytics YOLOv8. https://github.com/ultralytics/ultralytics

    2023

  15. [23]

    Tianjiao Li, Jun Liu, Wei Zhang, Yun Ni, Wenqian Wang, and Zhiheng Li. 2021. Uav-human: A large benchmark for human behavior under- standing with unmanned aerial vehicles. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition . 16266–16275

    2021

  16. [24]

    Yuanqi Li, Arthi Padmanabhan, Pengzhan Zhao, Yufei Wang, Guo- qing Harry Xu, and Ravi Netravali. 2020. Reducto: On-camera filtering for resource-efficient real-time video analytics. In Proceedings of the Annual conference of the ACM Special Interest Group on Data Commu- nicati...

    2020

  17. [25]

    Zechen Li, Shohreh Deldari, Linyao Chen, Hao Xue, and Flora D Salim

  18. [26]

    Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. 2024. On llms-driven synthetic data generation, curation, and evaluation: A survey. arXiv preprint arXiv:2406.15126 (2024)

    2024 arXiv

  19. [27]

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics. PMLR, 1273–1282

    2017

  20. [28]

    Vineet Mehta, Abhinav Dhall, Sujata Pal, and Shehroz S Khan. 2021. Motion and region aware adversarial learning for fall detection with thermal imaging. In 2020 25th international conference on pattern recog- nition (ICPR). IEEE, 6321–6328

    2021

  21. [29]

    arXiv preprint arXiv:2410.10624 (2024)

    Sensorllm: Aligning large language models with motion sensors for human activity recognition. arXiv preprint arXiv:2410.10624 (2024). 13 Preprint, TBD, TBD Siyang Jiang, Bufang Yang et al

    2024 arXiv

  22. [30]

    Xiaomin Ouyang, Xian Shuai, Yang Li, Li Pan, Xifan Zhang, Heming Fu, Sitong Cheng, Xinyan Wang, Shihua Cao, Jiang Xin, et al . 2024. ADMarker: A Multi-Modal Federated Learning System for Monitoring Digital Biomarkers of Alzheimer’s Disease. In Proceedings of the 30th Annual In...

    2024

  23. [31]

    Xiaomin Ouyang, Xian Shuai, Jiayu Zhou, Ivy Wang Shi, Zhiyuan Xie, Guoliang Xing, and Jianwei Huang. 2022. Cosmo: contrastive fusion learning with small data for multimodal human activity recognition. In Proceedings of the 28th Annual International Conference on Mobile Computi...

    2022

  24. [32]

    Xiaomin Ouyang and Mani Srivastava. 2024. LLMSense: Harnessing LLMs for high-level reasoning over spatiotemporal sensor traces. In 2024 IEEE 3rd Workshop on Machine Learning on Edge in Sensor Systems (SenSys-ML). IEEE, 9–14

    2024

  25. [33]

    Yiming Miao, Jun Yang, Bander Alzahrani, Guoguang Lv, Tarik Alafif, Ahmed Barnawi, and Min Chen. 2022. Abnormal behavior learning based on edge computing toward a crowd monitoring system. IEEE Network 36, 3 (2022), 90–96

    2022

  26. [34]

    Joseph Redmon and Ali Farhadi. 2017. YOLO9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7263–7271

    2017

  27. [35]

    Dominick Reilly and Srijan Das. 2024. Just add?! pose induced video transformers for understanding activities of daily living. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 18340–18350

    2024

  28. [36]

    Adrian Sanchez-Caballero, Sergio de López-Diz, David Fuentes- Jimenez, Cristina Losada-Gutiérrez, Marta Marrón-Romera, David Casillas-Perez, and Mohammad Ibrahim Sarker. 2022. 3dfcnn: Real- time action recognition using 3d deep neural networks with raw depth information. Multi...

    2022

  29. [37]

    Ajay Patel, Colin Raffel, and Chris Callison-Burch. 2024. Datadreamer: A tool for synthetic data generation and reproducible llm workflows. arXiv preprint arXiv:2402.10379 (2024)

    2024 arXiv

  30. [38]

    Nyle Siddiqui, Praveen Tirupattur, and Mubarak Shah. 2024. DVANet: Disentangling view and action features for multi-view action recog- nition. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 38. 4873–4881

    2024

  31. [39]

    Yue-meng Sun, Zhi-yun Wang, Yuan-yuan Liang, Chen-wei Hao, and Chang-he Shi. 2024. Digital biomarkers for precision diagnosis and monitoring in Parkinson’s disease. NPJ digital medicine 7, 1 (2024), 218

    2024

  32. [40]

    Zehua Sun, Qiuhong Ke, Hossein Rahmani, Mohammed Bennamoun, Gang Wang, and Jun Liu. 2022. Human action recognition from various data modalities: A review. IEEE transactions on pattern analysis and machine intelligence 45, 3 (2022), 3200–3225

    2022

  33. [41]

    Adrian Sanchez-Caballero, David Fuentes-Jimenez, and Cristina Losada-Gutiérrez. 2020. Exploiting the convlstm: Human action recog- nition using raw depth video-based recurrent neural networks. arXiv preprint arXiv:2006.07744 (2020)

    2020 arXiv

  34. [42]

    Md Touhiduzzaman, Steven M Hernandez, Peter E Pidcoe, and Eyuphan Bulut. 2025. Wi-PT-Hand: Wireless Sensing based Low- cost Physical Rehabilitation Tracking for Hand Movements. ACM Transactions on Computing for Healthcare 6, 1 (2025), 1–25

    2025

  35. [43]

    Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. 2024. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.00634 (2024)

    2024 arXiv

  36. [44]

    Yuxuan Weng, Guoquan Wu, Tianyue Zheng, Yanbing Yang, and Jun Luo. 2024. Large Model for Small Data: Foundation Model for Cross- Modal RF Human Activity Recognition. InProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems . 436–449

    2024

  37. [45]

    Zitian Tang, Wenjie Ye, Wei-Chiu Ma, and Hang Zhao. 2023. What Happened 3 Seconds Ago? Inferring the Past with Thermal Imaging. In CVPR

    2023

  38. [46]

    Zongxing Xie, Bing Zhou, Xi Cheng, Elinor Schoenfeld, and Fan Ye

  39. [47]

    Fuhai Xiong, Junxian Wang, Yushi Liu, Xudong Yan, Kamen Ivanov, Lei Wang, and Yan Yan. 2024. A Novel Human Activity Recognition Framework Based on Pre-Trained Foundation Model. In 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) . IEEE, 5712–5718

    2024

  40. [48]

    Huatao Xu, Liying Han, Qirui Yang, Mo Li, and Mani Srivastava. 2024. Penetrative ai: Making llms comprehend the physical world. InProceed- ings of the 25th International Workshop on Mobile Computing Systems and Applications. 1–7

    2024

  41. [49]

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang,...

    2024 arXiv

  42. [50]

    Bufang Yang, Yunqi Guo, Lilin Xu, Zhenyu Yan, Hongkai Chen, Guo- liang Xing, and Xiaofan Jiang. 2025. Socialmind: Llm-based proactive ar social assistive system with human-like perception for in-situ live interactions. Proceedings of the ACM on Interactive, Mobile, Wearable an...

    2025

  43. [51]

    In 2021 IEEE 9th International Conference on Healthcare Informatics (ICHI)

    Vitalhub: Robust, non-touch multi-user vital signs monitoring us- ing depth camera-aided uwb. In 2021 IEEE 9th International Conference on Healthcare Informatics (ICHI) . IEEE, 320–329

    2021

  44. [52]

    Bufang Yang, Lixing He, Kaiwei Liu, and Zhenyu Yan. 2024. Viassist: Adapting multi-modal large language models for users with visual impairments. In 2024 IEEE International Workshop on Foundation Models for Cyber-Physical Systems & Internet of Things (FMSys) . IEEE, 32–37

    2024

  45. [53]

    Bufang Yang, Siyang Jiang, Lilin Xu, Kaiwei Liu, Hai Li, Guoliang Xing, Hongkai Chen, Xiaofan Jiang, and Zhenyu Yan. 2024. Drhouse: An llm-empowered diagnostic reasoning system through harnessing outcomes from sensor data and expert knowledge. Proceedings of the ACM on Interac...

    2024

  46. [54]

    Lilin Xu, Chaojie Gu, Rui Tan, Shibo He, and Jiming Chen. 2023. Mesen: Exploit multimodal data to design unimodal human activity recog- nition with few labels. In Proceedings of the 21st ACM Conference on Embedded Networked Sensor Systems . 1–14

    2023

  47. [55]

    Liping Yuan, Jiawei Wang, Haomiao Sun, Yuchen Zhang, and Yuan Lin. 2025. Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding. arXiv preprint arXiv:2501.07888 (2025)

    2025 arXiv

  48. [56]

    Bufang Yang, Lixing He, Neiwen Ling, Zhenyu Yan, Guoliang Xing, Xian Shuai, Xiaozhe Ren, and Xin Jiang. 2023. Edgefm: Leveraging foundation model for open-set learning on the edge. In Proceedings of the 21st ACM Conference on Embedded Networked Sensor Systems . 111–124

    2023

  49. [57]

    Xiaotong Yuan and Ping Li. 2022. On convergence of fedprox: Local dissimilarity invariant bounds, non-smoothness and beyond.Advances in Neural Information Processing Systems 35 (2022), 10752–10765

    2022

  50. [58]

    Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. Scaling re- lationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825 (2023)

    2023 arXiv

  51. [59]

    14 Llambda Preprint, TBD, TBD

    Jiazuo Yu, Haomiao Xiong, Lu Zhang, Haiwen Diao, Yunzhi Zhuge, Lanqing Hong, Dong Wang, Huchuan Lu, You He, and Long Chen. 14 Llambda Preprint, TBD, TBD

  52. [60]

    arXiv preprint arXiv:2410.20178 (2024)

    Llms can evolve continually on modality for x-modal reasoning. arXiv preprint arXiv:2410.20178 (2024)

    2024 arXiv

  53. [61]

    Yuxin Zhang, Yiqiang Chen, Jindong Wang, and Zhiwen Pan. 2021. Unsupervised deep anomaly detection for multi-sensor time-series signals. IEEE Transactions on Knowledge and Data Engineering 35, 2 (2021), 2118–2132

    2021

  54. [62]

    Mu Yuan, Lan Zhang, Fengxiang He, Xueting Tong, and Xiang-Yang Li. 2022. InFi: End-to-end learnable input filter for resource-efficient mobile-centric inference. InProceedings of the 28th Annual International Conference on Mobile Computing And Networking . 228–241

    2022

  55. [63]

    Yanyi Zhang, Xinyu Li, and Ivan Marsic. 2021. Multi-label activity recognition using activity-specific features and activity correlations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14625–14635

    2021

  56. [64]

    Benjia Zhou, Pichao Wang, Jun Wan, Yanyan Liang, and Fan Wang

  57. [65]

    Haosong Zhang, Mei Chee Leong, Liyuan Li, and Weisi Lin. 2024. PeVL: Pose-Enhanced Vision-Language Model for Fine-Grained Hu- man Action Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 18857–18867

    2024

  58. [66]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations

    2019

  59. [68]

    Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. 2023. Meta-transformer: A unified framework for multimodal learning. arXiv preprint arXiv:2307.10802 (2023)

    2023 arXiv

  60. [71]

    IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 10 (2023), 11428–11442

    A unified multimodal de-and re-coupling framework for rgb-d motion recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 10 (2023), 11428–11442

    2023

  61. [72]

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al . 2023. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment.arXiv preprint arXiv:2310.01852 (2023). 15

    2023 arXiv

  62. [2021]

    In 2021 IEEE International Conference on Robotics and Automation (ICRA)

    A multi-spectral dataset for evaluating motion estimation sys- tems. In 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 5560–5566

    2021

  63. [2023]

    Advances in neural information processing systems 36 (2023), 10088–10115

    Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems 36 (2023), 10088–10115

    2023

  64. [2024]

    In 2024 IEEE 30th International Conference on Parallel and Distributed Systems (ICPADS)

    Towards llm-powered ambient sensor based multi-person human activity recognition. In 2024 IEEE 30th International Conference on Parallel and Distributed Systems (ICPADS) . IEEE, 609–616

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.