Pith. sign in

REVIEW 5 major objections 7 minor 49 references

Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model

T0 review · 5 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Period-LLM claims that multimodal LLMs can reason about periodic phenomena if trained easy-to-hard with an anti-forgetting gradient rule.

desk verdict A genuinely useful macro-periodic benchmark and training recipe, but the micro-periodic rPPG claim rests on an experimental design that cannot actually resolve a cardiac cycle. read the letter →

arxiv 2505.24476 v1 pith:HDSHYSMY submitted 2025-05-30 cs.CV

Yuting Zhang , Hao Lu , Qingyong Hu , Yin Wang , Kaishen Yuan , Xin Liu , Kaishun Wu This is my paper
classification cs.CV
keywords multimodallargelanguagemodelperiodictasksrepetitioncountingremotephotoplethysmographyeasy-to-hardgeneralizationlogicalforgettingvideoquestionansweringcross-modalreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the reason multimodal large language models fail at periodic tasks—counting repeated motions, estimating heart rate from a face video, predicting trajectory repetition—is a training pathology rather than a missing architectural capacity. The proposed remedy has two parts: an easy-to-hard curriculum that first has the model count repeated words in plain text, then fine-tunes it on macro video repetition counting and micro physiological-signal estimation, and a Resisting Logical Oblivion gradient-weighting rule that stops the later semantic-alignment stage from erasing the counting logic learned earlier. On newly constructed question-answer benchmarks, the resulting Period-LLM reports the best error among all compared models, with mean absolute error 3.77 on Countix-QA repetition counting versus 5.34 for the strongest fine-tuned baseline, and 13.78 on rPPG-QA heart-rate estimation versus 17.51. The paper also reports gains on rotated-digit, taxi-trajectory, and radar vital-sign question-answer tasks, supporting the claim that periodic reasoning transfers across modalities. If the recipe is correct, general multimodal models can acquire periodicity skills through cheap text pretraining plus fine-tuning, and the main obstacle is forgetting, not data scarcity.

What carries the argument

The two load-bearing objects are the easy-to-hard curriculum and the Resisting Logical Oblivion gradient weight. The curriculum moves from repeated-word counting in text, where the LLM is already strong, to macro video repetition counting (Countix-QA), then to micro physiological-signal estimation (rPPG-QA), and finally to auxiliary modalities such as rotated digits, taxi GPS trajectories, and radar vital signs; the stated rationale is that repetitiveness is the shared structure, so the model should learn it where the signal-to-noise ratio is highest. RLO is a per-channel gradient multiplier $\Omega(c_i) = 1 + \beta e^{\mathrm{iternum}/\mathrm{maxiter}}$ applied when a feature channel's mean activation is below the average across channels, and 1 otherwise, with $\beta=0.05$; it boosts updates to previously underused channels so new semantic knowledge lands there instead of overwriting the reasoning channels. Together these mechanisms implement the paper's framing of the task as a conflict between short-time semantic alignment and long-time periodic reasoning.

What would settle it

Feed the same 20 rPPG frames to the trained model with their temporal order reversed, or with a single frame repeated, and re-measure MAE on rPPG-QA; if the error stays near 13.78, the model is exploiting non-temporal statistics or textual priors rather than periodic signal. A second check is to fine-tune on Countix-QA without the text-counting pretraining and compare rPPG-QA MAE, which isolates whether the easy task or the macro-to-micro transfer is the actual carrier.

Watch

Extended reading notes

Core claim

The central claim is that periodic reasoning is a transferable skill that a multimodal large language model can learn once in the cheapest modality and carry into harder ones. Period-LLM is trained first on a generated text task in which it counts how many times a word appears in a repeated-word string; it is then fine-tuned on Countix-QA, a video repetition-counting set, and rPPG-QA, a face-video heart-rate estimation set assembled from five rPPG datasets. The central technical idea is that the later fine-tuning stages must not update all model parameters uniformly: RLO multiplies the backpropagated gradient of underused feature channels by a factor larger than one, so newly learned semantic information is absorbed into those channels while the channels encoding the earlier counting logic are left alone. With this recipe, the model beats all compared multimodal LLMs on every benchmark constructed in the paper, including Countix-QA MAE 3.77 versus the best baseline 5.34 and rPPG-QA MAE 13.78 versus 17.51, and the ablations attribute part of the gain to the easy text task and part to RLO.

Load-bearing premise

The heart-rate results assume that 20 uniformly sampled $224\times224$ frames from a face video contain enough of the periodic color-change signal to estimate heart rate, an assumption stated in Section 5.2 where every video input is reported to contain 20 frames.

Editorial extensions

If this is right

  • Pre-training on generated repeated-word questions improves zero-shot periodic reasoning on video and other modalities before any visual fine-tuning is done.
  • Fine-tuning on macro video repetition counting transfers to micro physiological-signal estimation: adding Countix-QA plus RLO lowers rPPG-QA MAE from 14.46 to 13.78 in the ablation.
  • The same model, after the two-stage recipe, produces the lowest errors among compared models on rotated-digit angle estimation, taxi-trajectory prediction, and radar vital-sign question answering, suggesting the periodic skill is modality-general.
  • Because RLO only changes the gradient weighting during fine-tuning, the recipe can be applied to an existing multimodal LLM without changing its architecture, inference cost, or deployment interface.
  • The new question-answer benchmarks isolate periodic reasoning from general video QA, giving future work a metric that separates counting and signal-estimation accuracy from caption quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An unpublished but testable implication is that the RLO gradient-gating trick is not specific to periodicity: the same protect-old-logic, absorb-new-semantics update rule should transfer to other numeric or logical skills, such as object counting or arithmetic word problems, when a model is adapted to a new domain.
  • The paper's motivating applications—weather forecasting, traffic flow, gait monitoring—are not among its benchmarks; a natural next experiment is to apply the same curriculum to explicit time-series modalities such as weather or traffic data.
  • A frame-order ablation would settle whether the reported heart-rate gain comes from temporal periodic reasoning or from the model learning the answer distribution of the aggregated rPPG datasets; that experiment is not in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper introduces Period-LLM, a multimodal large language model designed to improve performance on periodic and quasi-periodic tasks across text, video, and other modalities. The authors construct new QA benchmarks (Countix-QA, rPPG-QA, RotNIST-QA, Drive-QA, Radar-QA), propose an easy-to-hard training paradigm that starts with repetitive-text tasks and progresses to macro- and micro-periodic video tasks, and introduce a Resisting Logical Oblivion (RLO) gradient-weighting strategy to preserve periodic reasoning during semantic alignment. Experiments report that Period-LLM outperforms several existing MLLMs on these benchmarks, with ablations attributing gains to the easy-to-hard curriculum and RLO.

Significance. The problem of periodic reasoning in MLLMs is timely and underexplored, and the paper makes a useful first step by identifying the task family, constructing QA-style benchmarks, and open-sourcing code. If the empirical claims were fully supported, the easy-to-hard curriculum would be a plausible recipe for injecting temporal/periodic inductive biases into MLLMs. However, the current evidence is weakened by a questionable rPPG input protocol that likely cannot support true cardiac-cycle reasoning, an unfair zero-shot comparison, small and single-run ablation gains, and under-specified benchmark statistics. The paper does not currently demonstrate the claimed cross-modal generality.

major comments (5)
  1. [§5.2, Table 1] The rPPG-QA protocol uses 20 frames per video at 224×224 resolution. At typical video frame rates (25–30 fps), 20 consecutive frames span less than one cardiac cycle (the period is 0.6–1.0 s at 60–100 bpm), and uniform sampling across a longer video destroys the temporal resolution needed to resolve the quasi-periodic color signal. Under this protocol the input contains insufficient temporal information for heart-rate estimation from periodic chromatic changes, so the reported MAE of 13.78 bpm cannot be attributed to micro-periodic reasoning; the model may be relying on dataset statistics or textual priors. The paper should either use an rPPG-appropriate sampling (e.g., 128+ frames at the original frame rate) or provide a control experiment (e.g., shuffled frames or temporally downsampled input) to show that the periodic signal is actually exploited. Without this, the 'hard' stage of the easy-to-hard paradigm lacks supporting evidence.
  2. [§5.3, Table 1 and Table 2 captions] The term 'zero-shot' is used inequitably. In Table 1, other MLLMs make direct predictions with no training, whereas Period-LLM is 'pretrained on a text-QA dataset'; in Table 2, Period-LLM's 'zero-shot' results come from a model that was 'fine-tuned on the two visual periodic tasks from Table 1.' This is auxiliary-task pre-training or warm-starting, not zero-shot evaluation. The claimed zero-shot superiority is therefore confounded with the benefit of task-adjacent training. The authors should either train all baselines under the same pre-training protocol or relabel the setting (e.g., 'pre-trained on auxiliary periodic data') and discuss the comparison accordingly.
  3. [§5.5, Tables 3c and 3d] The ablation shows that removing RLO changes MAE from 3.77 to 3.89 on Countix-QA (0.12 improvement) and from 13.78 to 14.05 on rPPG-QA (0.27 improvement). All numbers are single-run and no error bars or significance tests are reported, so it is unclear whether the RLO effect exceeds run-to-run variability. Given that RLO is a core contribution ('Resisting Logical Oblivion'), the paper should report multiple seeds with standard deviations and ideally a significance test. The narrow β sweep in Table 3a (range 3.77–4.05) also suggests low sensitivity, which should be discussed in relation to the claimed mechanism.
  4. [§5.1, §5.3] The benchmark construction is under-specified. The paper does not report the number of test questions per dataset, the distribution of answer values (e.g., repetition counts, heart rates), or any human validation of the GPT-4-generated QA pairs. Without these statistics, the reader cannot assess whether the numerical answers are predictable from dataset priors (e.g., the mean count or heart rate) or whether the model is exploiting linguistic patterns rather than periodic reasoning. The authors should release the exact evaluation splits and include per-dataset answer distributions, along with a mean-predictor or majority-predictor baseline for each numeric task.
  5. [§4.3, Eq. (7)] The RLO weight function is not fully specified. The threshold condition uses c_i, 'the average of the ith feature channel,' but the paper does not state over which dimensions (token, spatial, batch) this average is computed, nor does it justify why below-average mean activation implies a 'redundant' channel. The weighting also grows with iteration number regardless of the gradient magnitude of the channel, so it is not directly tied to 'originally lower gradients' as described in the text. Please clarify the definition and provide a direct empirical link between the weighting rule and the observed MAE gains.
minor comments (7)
  1. [Abstract] There is a typo: 'Specially, We adopt' should be 'Specifically, we adopt'.
  2. [§6, Conclusion] The text 'alignment.the Our experimental results' contains a formatting error; it should read 'alignment. Our experimental results'.
  3. [§5.4, unnumbered appendix reference] The paper says 'More QA examples are provided in the appendix,' but the manuscript contains no appendix. Please either include the appendix or remove the reference.
  4. [References] Several references are duplicated: VideoChat appears as [13] and [14], VideoChatGPT as [29] and [30], Video-LLaMA as [47] and [48], and LLaVA as [17] and [18]. Please deduplicate and fix the citation numbers.
  5. [§4.3, Eq. (7)] The exponential term in Eq. (7) is ambiguous; please add parentheses to clarify the exponent, e.g., exp(iternum/maxiter).
  6. [Figure 7] The loss curves lack axis labels; the y-axis is labeled 'QA_loss' but there is no scale or training-step axis, making the convergence claim difficult to verify.
  7. [§5.1, RotNIST-QA] RotNIST-QA is a static image task (rotated digits), not a temporal periodic task; the paper should justify why rotational invariance is in a benchmark described as 'periodic tasks across various modalities.'

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the empirical claims are tested on held-out external benchmarks, and the cited prior work is background only.

full rationale

The paper's central claim is empirical rather than derivational: Period-LLM is trained on text-QA, Countix-QA, and rPPG-QA data, and evaluated against external MLLM baselines on held-out evaluation portions such as the V4V split for rPPG-QA. The easy-to-hard paradigm and the Resisting Logical Oblivion strategy are defined by the paper's own equations (Eqs. 5-9) and are not constructed from the evaluation target; the ablations in Tables 3c and 3d show incremental MAE changes from removing components. The GPT-4-generated text dataset is constructed from explicit repetition counts, so the text task is not a re-description of the video metrics. There are self-citations to earlier rPPG work by overlapping authors (e.g., Refs. [22] and [51]), but these appear as introductory background for micro-periodic signals and are not used to justify the reported performance improvements. The concern that 20 frames may be insufficient to resolve cardiac cycles is a plausible external-validity limitation of the rPPG-QA benchmark, but it does not establish that the reported MAE reduces by construction to a fitted parameter or to an input label; for that reason it is a correctness risk, not a circularity. Overall, the paper is self-contained against external benchmarks and no load-bearing circular step was identified.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical entities or forces. The central claim depends on two tuned hyperparameters (beta and threshold type) and three domain assumptions about task difficulty ordering and channel redundancy. The most fragile assumption is the RLO redundancy heuristic, which has no formal justification.

free parameters (2)
  • beta (RLO weight parameter) = 0.05
    Hyperparameter in the RLO weight function (Eq. 7), tuned on Countix-QA validation. Table 3a shows it affects MAE, with 0.05 giving the lowest value.
  • RLO threshold type = mean (channel average)
    The threshold (mean vs median vs learnable) determines which channels get up-weighted. Table 3b shows mean gives the best MAE (3.77), and it was selected for the main experiments.
assumptions (3)
  • domain assumption LLMs process text better than other modalities, so text is the easiest starting point for periodic tasks.
    Section 4.2.1 states that using text-only input simplifies the task by eliminating multimodal alignment. This underlies the easy-to-hard ordering of the curriculum.
  • domain assumption Macro-periodic video tasks (like repetition counting) are easier than micro-periodic tasks (like rPPG) because the periodic amplitude is larger relative to noise.
    Section 3 argues that the periodic amplitude K is larger for macro tasks, making periodicity easier to detect. This justifies the curriculum order from Countix to rPPG.
  • ad hoc to paper Feature channels with below-average mean activations are redundant and can be preferentially updated without harming existing knowledge.
    Section 4.3 and Eqs. 7-8: the RLO strategy assumes low-mean channels represent 'redundant feature channels' that can absorb new semantic information. No independent empirical or theoretical support is given for this channel-mean redundancy assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model." pith.science (2026). https://pith.science/paper/HDSHYSMY

@misc{pith2026250524476,
  author       = {Pith},
  title        = {Pith review of: Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HDSHYSMY}},
  note         = {Machine review of arXiv:2505.24476}
}
read the original abstract

Periodic or quasi-periodic phenomena reveal intrinsic characteristics in various natural processes, such as weather patterns, movement behaviors, traffic flows, and biological signals. Given that these phenomena span multiple modalities, the capabilities of Multimodal Large Language Models (MLLMs) offer promising potential to effectively capture and understand their complex nature. However, current MLLMs struggle with periodic tasks due to limitations in: 1) lack of temporal modelling and 2) conflict between short and long periods. This paper introduces Period-LLM, a multimodal large language model designed to enhance the performance of periodic tasks across various modalities, and constructs a benchmark of various difficulty for evaluating the cross-modal periodic capabilities of large models. Specially, We adopt an "Easy to Hard Generalization" paradigm, starting with relatively simple text-based tasks and progressing to more complex visual and multimodal tasks, ensuring that the model gradually builds robust periodic reasoning capabilities. Additionally, we propose a "Resisting Logical Oblivion" optimization strategy to maintain periodic reasoning abilities during semantic alignment. Extensive experiments demonstrate the superiority of the proposed Period-LLM over existing MLLMs in periodic tasks. The code is available at https://github.com/keke-nice/Period-LLM.

Figures

Figures reproduced from arXiv: 2505.24476 by the authors.

Figure 1
Figure 1. Existing multimodal models may fail to analyze periodic tasks, such as motion counting, traffic flow, weather forecasting, and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The process of the generation of Question-and-answer datasets. Samples of text and video are taken as examples. In the text [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The pipeline of training paradigm from easy to hard. The model begins training with simpler periodic tasks and gradually [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The Period-LLM framework, using video as an example, integrates both text and video inputs. The video is processed through a [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Details of the function of the gradient weight function. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Examples from Countix-QA and rPPG-QA. Period-LLM (with text-QA pretraining and RLO) Baseline QA_loss [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Comparison of Loss Reduction Between Baseline [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 27 canonical work pages

  1. [1]

    Weather forecasting model using artificial neural network

    Kumar Abhishek, Maheshwari Prasad Singh, Saswata Ghosh, and Abhishek Anand. Weather forecasting model using artificial neural network. Procedia Technology, 4:311– 318, 2012. 1

    work page 2012

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  3. [3]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  4. [4]

    Spatio-temporal atten- tion and magnification for classification of parkinson’s dis- ease from videos collected via the internet

    Mohammad Rafayet Ali, Javier Hernandez, E Ray Dorsey, Ehsan Hoque, and Daniel McDuff. Spatio-temporal atten- tion and magnification for classification of parkinson’s dis- ease from videos collected via the internet. In 2020 15th IEEE International Conference on Automatic Face and Ges- ture Recognition (FG 2020), pages 207–214. IEEE, 2020. 3

    work page 2020

  5. [5]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G ¨ul Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021. 3

    work page 2021

  6. [6]

    Unsupervised skin tissue seg- mentation for remote photoplethysmography.Pattern Recog- nition Letters, 124:82–90, 2019

    Serge Bobbia, Richard Macwan, Yannick Benezeth, Alamin Mansouri, and Julien Dubois. Unsupervised skin tissue seg- mentation for remote photoplethysmography.Pattern Recog- nition Letters, 124:82–90, 2019. 6

    work page 2019

  7. [7]

    Counting out time: Class agnostic video repetition counting in the wild

    Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Counting out time: Class agnostic video repetition counting in the wild. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10387–10396, 2020. 1, 2, 3, 6

    work page 2020

  8. [8]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023. 2

    2023

Show all 49 references
  1. [9]

    Onellm: One framework to align all modalities with language

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xi- angyu Yue. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26584–...

    2024

  2. [10]

    Transrac: Encoding multi- scale temporal correlation with transformers for repetitive action counting

    Huazhang Hu, Sixun Dong, Yiqun Zhao, Dongze Lian, Zhengxin Li, and Shenghua Gao. Transrac: Encoding multi- scale temporal correlation with transformers for repetitive action counting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages ...

    2022

  3. [11]

    Gradient-based learning applied to document recog- nition

    Yann LeCun, L ´eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recog- nition. Proceedings of the IEEE , 86(11):2278–2324, 1998. 6

    1998

  4. [12]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. 2

    2023 arXiv

  5. [14]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 6, 7

    2023 arXiv

  6. [15]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–...

    2024

  7. [16]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.17043, 2023. 2, 6, 7

    2023 arXiv

  8. [17]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2

    2024

  9. [18]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2, 6

    2024

  10. [19]

    One for all: Video conversation is fea- sible without video instruction tuning

    Ruyang Liu, Chen Li, Yixiao Ge, Ying Shan, Thomas H Li, and Ge Li. One for all: Video conversation is fea- sible without video instruction tuning. arXiv preprint arXiv:2309.15785, 2023. 6, 7

    2023 arXiv

  11. [20]

    Multi-task temporal shift attention networks for on-device contactless vitals measurement

    Xin Liu, Josh Fromm, Shwetak Patel, and Daniel McDuff. Multi-task temporal shift attention networks for on-device contactless vitals measurement. Advances in Neural Infor- mation Processing Systems, 33:19400–19411, 2020. 3

    2020

  12. [21]

    Metaphys: few-shot adaptation for non-contact physiological measurement

    Xin Liu, Ziheng Jiang, Josh Fromm, Xuhai Xu, Shwetak Patel, and Daniel McDuff. Metaphys: few-shot adaptation for non-contact physiological measurement. In Proceedings of the conference on health, inference, and learning , pages 154–163, 2021. 3

    2021

  13. [22]

    rppg-mae: Self-supervised pretraining with masked autoencoders for remote physiological mea- surements

    Xin Liu, Yuting Zhang, Zitong Yu, Hao Lu, Huanjing Yue, and Jingyu Yang. rppg-mae: Self-supervised pretraining with masked autoencoders for remote physiological mea- surements. IEEE Transactions on Multimedia, 2024. 1

    2024

  14. [23]

    Monitoring gait at home with radio waves in parkin- son’s disease: A marker of severity, progression, and med- ication response

    Yingcheng Liu, Guo Zhang, Christopher G Tarolli, Rumen Hristov, Stella Jensen-Roberts, Emma M Waddell, Taylor L Myers, Meghan E Pawlik, Julia M Soto, Renee M Wilson, et al. Monitoring gait at home with radio waves in parkin- son’s disease: A marker of severity, progression, an...

    2022

  15. [24]

    Class-agnostic counting

    Erika Lu, Weidi Xie, and Andrew Zisserman. Class-agnostic counting. In Computer Vision–ACCV 2018: 14th Asian Con- ference on Computer Vision, Perth, Australia, December 2– 6, 2018, Revised Selected Papers, Part III 14 , pages 669–

    2018

  16. [25]

    Dual-gan: Joint bvp and noise modeling for remote physiological measurement

    Hao Lu, Hu Han, and S Kevin Zhou. Dual-gan: Joint bvp and noise modeling for remote physiological measurement. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 12404–12413, 2021. 1, 3

    2021

  17. [26]

    Gpt as psychologist? preliminary evaluations for gpt-4v on visual affective computing

    Hao Lu, Xuesong Niu, Jiyao Wang, Yin Wang, Qingyong Hu, Jiaqi Tang, Yuting Zhang, Kaishen Yuan, Bin Huang, Zi- tong Yu, et al. Gpt as psychologist? preliminary evaluations for gpt-4v on visual affective computing. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  18. [27]

    Valley: Video assistant with large language model enhanced ability

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207, 2023. 3

    2023 arXiv

  19. [28]

    Video anomaly detection and explanation via large language models

    Hui Lv and Qianru Sun. Video anomaly detection and explanation via large language models. arXiv preprint arXiv:2401.05702, 2024. 3

    2024 arXiv

  20. [30]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 6, 7

    2023 arXiv

  21. [31]

    Rhythmnet: End-to-end heart rate estimation from face via spatial-temporal representation

    Xuesong Niu, Shiguang Shan, Hu Han, and Xilin Chen. Rhythmnet: End-to-end heart rate estimation from face via spatial-temporal representation. IEEE Transactions on Im- age Processing, 29:2409–2423, 2019. 3, 6

    2019

  22. [32]

    Teaching clip to count to ten

    Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 3170–3180, 2023. 2

    2023

  23. [33]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  24. [34]

    The first vision for vitals (v4v) chal- lenge for non-contact video-based physiological estimation

    Ambareesh Revanur, Zhihua Li, Umur A Ciftci, Lijun Yin, and L ´aszl´o A Jeni. The first vision for vitals (v4v) chal- lenge for non-contact video-based physiological estimation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2760–2767, 2021. 6

    2021

  25. [35]

    Deepdownscale: A deep learning strategy for high-resolution weather forecast

    Eduardo Rocha Rodrigues, Igor Oliveira, Renato Cunha, and Marco Netto. Deepdownscale: A deep learning strategy for high-resolution weather forecast. In2018 IEEE 14th Interna- tional Conference on e-Science (e-Science), pages 415–422. IEEE, 2018. 1

    2018

  26. [36]

    A dataset of clinically recorded radar vital signs with synchro- nised reference sensor signals

    Sven Schellenberger, Kilin Shi, Tobias Steigleder, Anke Malessa, Fabian Michler, Laura Hameyer, Nina Neumann, Fabian Lurz, Robert Weigel, Christoph Ostgathe, et al. A dataset of clinically recorded radar vital signs with synchro- nised reference sensor signals. Scientific data...

  27. [37]

    Predicting weather forecast uncertainty with machine learning

    Sebastian Scher and Gabriele Messori. Predicting weather forecast uncertainty with machine learning. Quarterly Jour- nal of the Royal Meteorological Society , 144(717):2830– 2841, 2018. 1

    2018

  28. [38]

    Metnet: A neural weather model for precipitation forecasting

    Casper Kaae Sønderby, Lasse Espeholt, Jonathan Heek, Mostafa Dehghani, Avital Oliver, Tim Salimans, Shreya Agrawal, Jason Hickey, and Nal Kalchbrenner. Metnet: A neural weather model for precipitation forecasting. arXiv preprint arXiv:2003.12140, 2020. 3

    2003 arXiv

  29. [39]

    Deepphase: Periodic autoencoders for learning motion phase manifolds

    Sebastian Starke, Ian Mason, and Taku Komura. Deepphase: Periodic autoencoders for learning motion phase manifolds. ACM Transactions on Graphics (TOG), 41(4):1–13, 2022. 3

    2022

  30. [40]

    Non-contact video-based pulse rate measurement on a mo- bile service robot

    Ronny Stricker, Steffen M ¨uller, and Horst-Michael Gross. Non-contact video-based pulse rate measurement on a mo- bile service robot. In The 23rd IEEE International Sym- posium on Robot and Human Interactive Communication , pages 1056–1062. IEEE, 2014. 6

    2014

  31. [41]

    Pandagpt: One model to instruction-follow them all

    Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16355, 2023. 3

    2023 arXiv

  32. [42]

    Towards efficient and comprehen- sive urban spatial-temporal prediction: A unified library and performance benchmark

    Jingyuan Wang, Jiawei Jiang, Wenjun Jiang, Chengkai Han, and Wayne Xin Zhao. Towards efficient and comprehen- sive urban spatial-temporal prediction: A unified library and performance benchmark. arXiv preprint arXiv:2304.14343,

    arXiv

  33. [43]

    Image enhancement for remote photoplethys- mography in a low-light environment

    Lin Xi, Weihai Chen, Changchen Zhao, Xingming Wu, and Jianhua Wang. Image enhancement for remote photoplethys- mography in a low-light environment. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pages 1–7. IEEE, 2020. 6

    2020

  34. [44]

    Zero-shot video question answering via frozen bidirectional language models

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. Advances in Neural Information Processing Systems, 35:124–141, 2022. 6, 7

    2022

  35. [45]

    Simper: Simple self-supervised learning of periodic targets

    Yuzhe Yang, Xin Liu, Jiang Wu, Silviu Borac, Dina Katabi, Ming-Zher Poh, and Daniel McDuff. Simper: Simple self-supervised learning of periodic targets. arXiv preprint arXiv:2210.03115, 2022. 1

    2022 arXiv

  36. [46]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. 3

    2023 arXiv

  37. [48]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 2, 6, 7

    2023 arXiv

  38. [49]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention

    Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Ao- jun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023. 6, 7

    2023 arXiv

  39. [50]

    Repeti- tive activity counting by sight and sound

    Yunhua Zhang, Ling Shao, and Cees GM Snoek. Repeti- tive activity counting by sight and sound. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14070–14079, 2021. 1

    2021

  40. [51]

    Advancing generalizable remote physiological measurement through the integration of explicit and implicit prior knowledge

    Yuting Zhang, Hao Lu, Xin Liu, Yingcong Chen, and Kaishun Wu. Advancing generalizable remote physiological measurement through the integration of explicit and implicit prior knowledge. arXiv preprint arXiv:2403.06947, 2024. 1

    2024 arXiv

  41. [52]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 2

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.