V2N uses a one-second keyboard video window and four jointly trained heads to predict onset, offset, key hold, and velocity, achieving state-of-the-art visual piano transcription on PianoVAM and R3.
Virtual Piano using Computer Vision
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
In this research, Piano performances have been analyzed only based on visual information. Computer vision algorithms, e.g., Hough transform and binary thresholding, have been applied to find where the keyboard and specific keys are located. At the same time, Convolutional Neural Networks(CNNs) has been also utilized to find whether specific keys are pressed or not, and how much intensity the keys are pressed only based on visual information. Especially for detecting intensity, a new method of utilizing spatial, temporal CNNs model is devised. Early fusion technique is especially applied in temporal CNNs architecture to analyze hand movement. We also make a new dataset for training each model. Especially when finding an intensity of a pressed key, both of video frames and their optical flow images are used to train models to find effectiveness.
citation-role summary
background 1
citation-polarity summary
fields
cs.SD 1years
2026 1verdicts
CONDITIONAL 1roles
background 1polarities
background 1representative citing papers
citing papers explorer
-
Multi-Task Multi-Frame Visual Piano Transcription
V2N uses a one-second keyboard video window and four jointly trained heads to predict onset, offset, key hold, and velocity, achieving state-of-the-art visual piano transcription on PianoVAM and R3.